edu.ucla.sspace.tools

Class TokenCounter

java.lang.Object

edu.ucla.sspace.tools.TokenCounter

public class TokenCounter
extends Object

A utility class for counting tokens in one or more files. This class also supports counting compound token instances, as well as counting for only a subset of the unique tokens. This class is intended for token counting in very large corpora where space-efficiency is important. The output is equivalent to the command cat corpus.txt | awk '{ split($0,a); for (i in a) { print a[i]; }}' | uniq -c. However, this command is significantly more memory and CPU intensive.

Author:

David Jurgens

Constructor Summary

Constructors

Constructor and Description
TokenCounter() Creates a new token counter
TokenCounter(boolean doLowerCasing) Creates a new token counter that optionally lower cases tokens

Method Summary

Methods

Modifier and Type	Method and Description
Map<String,Integer>	getTokenCounts() Returns a mapping from each seen token to the number of times it occurred
static void	main(String[] args)
void	process(BufferedReader br) Counts all of the tokens in the reader
void	process(String tokens) Counts all of the tokens in the string
void	processFile(File file) Counts all of the tokens in the file
void	processFile(String fileName) Counts all of the tokens in the file with specified name

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenCounter

public TokenCounter()

Creates a new token counter

TokenCounter

public TokenCounter(boolean doLowerCasing)

Creates a new token counter that optionally lower cases tokens

Parameters:

doLowerCasing - true if the token counter should lower case all tokens before counting

Method Detail

getTokenCounts

public Map<String,Integer> getTokenCounts()

Returns a mapping from each seen token to the number of times it occurred

processFile

public void processFile(String fileName)
                 throws IOException

Counts all of the tokens in the file with specified name

Throws:

IOException

processFile

public void processFile(File file)
                 throws IOException

Counts all of the tokens in the file

Throws:

IOException

process

public void process(BufferedReader br)

Counts all of the tokens in the reader

process

public void process(String tokens)

Counts all of the tokens in the string

main

public static void main(String[] args)