edu.ucla.sspace.text

Class CompoundWordIterator

java.lang.Object

edu.ucla.sspace.text.CompoundWordIterator

All Implemented Interfaces:

Iterator<String>

public class CompoundWordIterator
extends Object
implements Iterator<String>

An iterator over all the tokens in a stream, which supports tokenizing predetermined n-grams as single tokens. Compound tokenizing is performed greedily so the longest possible match from any tokenizing will be returned.

Note that unlike other iterators, the next method is O(n) in complexity where n is the number of unique words starting the set of compound tokens recognized by this iterator. This may result in a noticeable performance penalty when a large set of compound words is used.

This class also provides a reset method to allow resetting the token stream backing this iterator. If the recognized set of compound words does not change, then this method is prefered over creating a new CompoundWordIterator as it avoids the initializationg overhead of building the underlying compound recognizer.

Author:

David Jurgens

Constructor Summary

Constructors

Constructor and Description
CompoundWordIterator(BufferedReader br, Set<String> compoundWords)
CompoundWordIterator(Iterator<String> tokens, Set<String> compoundWords)
CompoundWordIterator(String str, Set<String> compoundWords)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	hasNext() Returns true if there is another token to return.
String	next() Returns the next token in the stream.
void	remove() Throws an UnsupportedOperationException if called.
void	reset(BufferedReader br) Resets the underlying token stream to point to the contents of the provided BufferedReader.
void	reset(Iterator<String> tokens) Resets the underlying token stream to point to the tokens in the provided iterator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompoundWordIterator

public CompoundWordIterator(String str,
                    Set<String> compoundWords)

CompoundWordIterator

public CompoundWordIterator(BufferedReader br,
                    Set<String> compoundWords)

CompoundWordIterator

public CompoundWordIterator(Iterator<String> tokens,
                    Set<String> compoundWords)

Method Detail

hasNext

public boolean hasNext()

Returns true if there is another token to return.

Specified by:

hasNext in interface Iterator<String>

next

public String next()

Returns the next token in the stream. Note that unlike other iterators, this method is O(n) in complexity where n is the number of unique words starting the set of compound tokens recognized by this iterator.

Specified by:

next in interface Iterator<String>

remove

public void remove()

Throws an UnsupportedOperationException if called.

Specified by:

remove in interface Iterator<String>

reset

public void reset(BufferedReader br)

Resets the underlying token stream to point to the contents of the provided BufferedReader. This does not change the set of accepted compound words.

reset

public void reset(Iterator<String> tokens)

Resets the underlying token stream to point to the tokens in the provided iterator. This does not change the set of accepted compound words.