edu.ucla.sspace.tools

Class DepTokenCounter

java.lang.Object

edu.ucla.sspace.tools.DepTokenCounter

public class DepTokenCounter
extends Object

A utility class for counting tokens in one or more files. This class also supports counting compound token instances, as well as counting for only a subset of the unique tokens. This class is intended for token counting in very large corpora where space-efficiency is important. The output is equivalent to the command cat corpus.txt | awk '{ split($0,a); for (i in a) { print a[i]; }}' | uniq -c. However, this command is significantly more memory and CPU intensive.

Author:

Keith Stevens

Constructor Summary

Constructors

Constructor and Description
DepTokenCounter(boolean doLowerCasing, boolean doPos, DependencyExtractor extractor) Creates a new token counter that optionally lower cases tokens

Method Summary

Methods

Modifier and Type	Method and Description
Map<String,Integer>	getTokenCounts() Returns a mapping from each seen token to the number of times it occurred
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DepTokenCounter

public DepTokenCounter(boolean doLowerCasing,
               boolean doPos,
               DependencyExtractor extractor)

Creates a new token counter that optionally lower cases tokens

Parameters:

doLowerCasing - true if the token counter should lower case all tokens before counting

Method Detail

getTokenCounts

public Map<String,Integer> getTokenCounts()

Returns a mapping from each seen token to the number of times it occurred

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception