edu.ucla.sspace.evaluation

Class AbstractWordAssociationTest

java.lang.Object

edu.ucla.sspace.evaluation.AbstractWordAssociationTest

All Implemented Interfaces:

WordAssociationTest

Direct Known Subclasses:

DeeseAntonymEvaluation

public abstract class AbstractWordAssociationTest
extends Object
implements WordAssociationTest

A test-runner for evaluating the performance of a SemanticSpace on a WordSimilarityEvaluation test.

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<Pair<String>,Double>	wordPairToHumanJudgement A mapping from a word pair to the human association judgement for it

Constructor Summary

Constructors

Constructor and Description
AbstractWordAssociationTest(Map<Pair<String>,Double> wordPairToHumanJudgement)

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract Double	computeAssociation(SemanticSpace sspace, String word1, String word2) Returns the association of the two words on a scale of 0 to 1.
protected double	computeScore(double[] humanScores, double[] compScores) Returns the correlation between the computer generated scores and the human evaluated scores.
WordAssociationReport	evaluate(SemanticSpace sspace) Evaluates the performance of a given SemanticSpace on a given WordSimilarityEvaluation using the provided similarity metric.
protected abstract double	getHighestScore() Returns the highest score possible for human judgments.
protected abstract double	getLowestScore() Returns the lowest score possible for human judgments.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

wordPairToHumanJudgement

protected final Map<Pair<String>,Double> wordPairToHumanJudgement

A mapping from a word pair to the human association judgement for it

Constructor Detail

AbstractWordAssociationTest

public AbstractWordAssociationTest(Map<Pair<String>,Double> wordPairToHumanJudgement)

Parameters:

wordPairToHumanJudgement - A mapping from a word pair to the human association judgement for it

Method Detail

evaluate

public WordAssociationReport evaluate(SemanticSpace sspace)

Evaluates the performance of a given SemanticSpace on a given WordSimilarityEvaluation using the provided similarity metric. Returns a WordSimilarityReport detailing the performance, with similarity scores scaled by the lowest and highest human based similarity ratings.

Specified by:

evaluate in interface WordAssociationTest

Parameters:

sspace - The SemanticSpace to test against

Returns:

A WordSimilarityReport detailing the performance

computeScore

protected double computeScore(double[] humanScores,
                  double[] compScores)

Returns the correlation between the computer generated scores and the human evaluated scores. Sub-classes can override this if the correlation metric is not suitable for the data set. Possible alternatives are mean square error or simply the average computer generated score.

getLowestScore

protected abstract double getLowestScore()

Returns the lowest score possible for human judgments. This score is interpreted as the least associated.

getHighestScore

protected abstract double getHighestScore()

Returns the highest score possible for human judgments. This score is interpreted as the most associated.

computeAssociation

protected abstract Double computeAssociation(SemanticSpace sspace,
                        String word1,
                        String word2)

Returns the association of the two words on a scale of 0 to 1. Subclasses should override this method to provide specific ways of determining the association of two words in the semantic space, but should ensure that the return value falls with the predefined scale.

Returns:

the assocation or null if either word1 or word2 are not in the semantic space