edu.ucla.sspace.tools

Class WikipediaCleaner

java.lang.Object

edu.ucla.sspace.tools.WikipediaCleaner

public class WikipediaCleaner
extends Object

A tool for converting Wikipedia Snapshots into a parsable corpus of documents.

Author:

David Jurgens, Keith Stevens

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WikipediaCleaner.CleanerOption

Constructor Summary

Constructors

Constructor and Description
WikipediaCleaner(String outputFile, Set<WikipediaCleaner.CleanerOption> options, int minTokensPerArticle) Create a new WikipediaCleaner which will read articles from outputFileName, with the given thresholds for link requirements.

Method Summary

Methods

Modifier and Type	Method and Description
static void	main(String[] args)
void	processDocument(edu.ucla.sspace.tools.WikipediaCleaner.WikiDoc doc) Process the content of the given WikiDoc.
void	removeExternalLinkMarkup(StringBuilder article) Replace [link] tags with link name and track what articles this article links to.
void	removeWikiLinkMarkup(StringBuilder article, String title) Replace [[link]] tags with link name and track what articles this article links to.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WikipediaCleaner

public WikipediaCleaner(String outputFile,
                Set<WikipediaCleaner.CleanerOption> options,
                int minTokensPerArticle)

Create a new WikipediaCleaner which will read articles from outputFileName, with the given thresholds for link requirements.

Method Detail

processDocument

public void processDocument(edu.ucla.sspace.tools.WikipediaCleaner.WikiDoc doc)

Process the content of the given WikiDoc.

Parameters:

doc - The WikiDoc to process.

removeWikiLinkMarkup

public void removeWikiLinkMarkup(StringBuilder article,
                        String title)

Replace [[link]] tags with link name and track what articles this article links to.

Parameters:

text - The article text to clean and process link structure of.

removeExternalLinkMarkup

public void removeExternalLinkMarkup(StringBuilder article)

Replace [link] tags with link name and track what articles this article links to.

Parameters:

text - The article text to clean and process link structure of.

main

public static void main(String[] args)