edu.ucla.sspace.text

Class DocumentPreprocessor

java.lang.Object

edu.ucla.sspace.text.DocumentPreprocessor

public class DocumentPreprocessor
extends Object

A class for preprocessing all types of documents. This approach was used by Rohde et al. (2004) for processing USENET articles.

Constructor Summary

Constructors

Constructor and Description
DocumentPreprocessor() Constructs a DocumentPreprocessor with an empty word list
DocumentPreprocessor(File wordList) Constructs a DocumentPreprocessor where the provided file contains the list all valid words for the output documents.
DocumentPreprocessor(String[] wordList) A Constructor purely for test purposes.

Method Summary

Methods

Modifier and Type	Method and Description
String	process(String document) Processes the provided document and returns the cleaned version of the document.
String	process(String document, boolean removeWords) Processes the provided document and returns the cleaned version of the document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DocumentPreprocessor

public DocumentPreprocessor()

Constructs a DocumentPreprocessor with an empty word list

DocumentPreprocessor

public DocumentPreprocessor(File wordList)
                     throws IOException

Constructs a DocumentPreprocessor where the provided file contains the list all valid words for the output documents.

Parameters:

wordList - a file containing a list of all valid words for outputting

Throws:

IOException

DocumentPreprocessor

public DocumentPreprocessor(String[] wordList)

A Constructor purely for test purposes. Pass in an array of words which will serve the roll as a word list.

Method Detail

process

public String process(String document)

Processes the provided document and returns the cleaned version of the document.

Parameters:

document - a document to process

Returns:

a cleaned version of the document

process

public String process(String document,
             boolean removeWords)

Processes the provided document and returns the cleaned version of the document.

Parameters:

document - a document to process

removeWords - If true, any word which is not found in the provided word list is removed from the cleaned document.

Returns:

a cleaned version of the document