gov.llnl.ontology.text.tokenize
Class StemmingTokenizer

java.lang.Object
  gov.llnl.ontology.text.tokenize.TokenizerAdaptor
      gov.llnl.ontology.text.tokenize.StemmingTokenizer

All Implemented Interfaces:

opennlp.tools.tokenize.Tokenizer

public class StemmingTokenizer
extends TokenizerAdaptor

A decorator for stemming each token returned by an existing Tokenizer, given a Stemmer.

Author:

Keith Stevens

Field Summary

Fields inherited from class gov.llnl.ontology.text.tokenize.TokenizerAdaptor

tokenizer

Constructor Summary

StemmingTokenizer(opennlp.tools.tokenize.Tokenizer tokenizer, edu.ucla.sspace.text.Stemmer stemmer)
Creates a new StemmingTokenizer that decorates the existing Tokenizer.

Method Summary

String[]

tokenize(String sentence)

Methods inherited from class gov.llnl.ontology.text.tokenize.TokenizerAdaptor

tokenizePos

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StemmingTokenizer

public StemmingTokenizer(opennlp.tools.tokenize.Tokenizer tokenizer,
                         edu.ucla.sspace.text.Stemmer stemmer)

Creates a new StemmingTokenizer that decorates the existing Tokenizer.

Parameters:

tokenizer - A Tokenizer to decorate

stemmer - The Stemmer used to stem each token

Method Detail

tokenize

public String[] tokenize(String sentence)

Specified by:

tokenize in interface opennlp.tools.tokenize.Tokenizer

Overrides:

tokenize in class TokenizerAdaptor