edu.cmu.sphinx.decoder.search
Class Token

java.lang.Object
  extended byedu.cmu.sphinx.decoder.search.Token
All Implemented Interfaces:
Scoreable
Direct Known Subclasses:
CombineToken, ParallelToken

public class Token
extends java.lang.Object
implements Scoreable

Represents a single state in the recognition trellis. Subclasses of a token are used to represent the various emitting state. All scores are maintained in LogMath log base


Field Summary
static java.util.Comparator COMPARATOR
          a token comparator that is used to order tokens in descending order
 
Constructor Summary
Token(float logAcousticScore, float logLanguageScore, Token predecessor)
          Creates a Token with the given acoustic and language scores and predecessor.
Token(SearchState state, int frameNumber)
          Creates the initial token with the given word history depth
 
Method Summary
 float calculateScore(Data feature, boolean keepData, float gain)
          Calculates a score against the given feature.
 Token child(SearchState state, float logTotalScore, float logLanguageScore, float logInsertionProbability, int frameNumber)
          Constructs a new token that continues the search from the current token.
 void dumpTokenPath()
          dumps a branch of tokens
 void dumpTokenPath(boolean includeHMMStates)
          dumps a branch of tokens
 float getAcousticScore()
          Returns the acoustic score for this token (in logMath log base)
 java.lang.Object getAppObject()
          Returns the application object
 Data getData()
          Returns the feature for this Token.
 int getFrameNumber()
          Returns the frame number for this token.
 float getInsertionProbability()
          Returns the insertionPenalty associated with this token
 float getLanguageScore()
          Returns the language score associated with this token
 int getLocation()
          Returns the location of this Token in the ActiveList.
 Token getPredecessor()
          Returns the predecessor for this token, or null if this token has no predecessors
 float getScore()
          Returns the score for the token.
 SearchState getSearchState()
          Returns the SearchState associated with this token
 Word getWord()
          Returns the word of this Token, the search state is a WordSearchState.
 java.lang.String getWordPath()
          Returns the string of words for this token, with embedded silences
 java.lang.String getWordPath(boolean wantFiller)
          Returns the string of words leading up to this token.
 java.lang.String getWordPathNoFiller()
          Returns the string of words for this token, with no embedded filler words
 java.lang.String getWordUnitPath()
          Returns the string of words and units for this token, with embedded silences.
 float getWorkingScore()
          Gets the working score.
 boolean isEmitting()
          Determines if this token is associated with an emitting state.
 boolean isFinal()
          Determines if this token is associated with a final SentenceHMM state.
 boolean isWord()
          Determines if this token marks the end of a word
 float normalizeScore(float maxLogScore)
          Normalizes a previously calculated score
 void setAppObject(java.lang.Object obj)
          Sets the application object
 void setLocation(int location)
          Sets the location of this Token in the ActiveList.
static void setPredecessorClass(java.util.Set bpClasses)
          Set the predecessor class.
 void setScore(float logScore)
          Sets the score for this token
 void setWorkingScore(float logScore)
          Sets the working score for this token
static void showCount()
          Shows the token count
 java.lang.String toString()
          Retrieves the string representation of this object
 boolean validate()
          Determines if this branch is valid
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

COMPARATOR

public static final java.util.Comparator COMPARATOR
a token comparator that is used to order tokens in descending order

Constructor Detail

Token

public Token(SearchState state,
             int frameNumber)
Creates the initial token with the given word history depth

Parameters:
state - the SearchState associated with this token
frameNumber - the frame number for this token

Token

public Token(float logAcousticScore,
             float logLanguageScore,
             Token predecessor)
Creates a Token with the given acoustic and language scores and predecessor.

Parameters:
logAcousticScore - the log acoustic score
logLanguageScore - the log language score
predecessor - the predecessor Token
Method Detail

setPredecessorClass

public static void setPredecessorClass(java.util.Set bpClasses)
Set the predecessor class. Used to modify the behavior of child() so that the predecessor backpointer will skip internal states. For example, when retaining tokens to form a word lattice, it would be inefficient to keep any states but WordStates-- other types of states are not used and the memory should be saved. On the other hand, a phoneme recognizer would require PronunciationStates to create a suitable result.

Parameters:
bpClasses -

child

public Token child(SearchState state,
                   float logTotalScore,
                   float logLanguageScore,
                   float logInsertionProbability,
                   int frameNumber)
Constructs a new token that continues the search from the current token. If predessorClasses is null or if the class of the state is a member of predecessorClasses, the predecessor of the new token is set to the current token. Otherwise it is set to the predecessor of the current token. This behavior is used to save memory when building lattices.

Parameters:
state - the SentenceHMMState associated with this token
logTotalScore - the total entry score for this token (in LogMath log base)
logLanguageScore - the language score associated with this token (in LogMath log base)
logInsertionProbability - the insertion probabilty associated with this token (in LogMath log base)
frameNumber - the frame number associated with this token

getPredecessor

public Token getPredecessor()
Returns the predecessor for this token, or null if this token has no predecessors

Returns:
the predecessor

getFrameNumber

public int getFrameNumber()
Returns the frame number for this token. Note that for tokens that are associated with non-emitting states, the frame number represents the next frame number. For emitting states, the frame number represents the current frame number.

Specified by:
getFrameNumber in interface Scoreable
Returns:
the frame number for this token

getData

public Data getData()
Returns the feature for this Token.

Returns:
the feature for this Token

getScore

public float getScore()
Returns the score for the token. The score is a combination of language and acoustic scores

Specified by:
getScore in interface Scoreable
Returns:
the score of this frame (in logMath log base)

calculateScore

public float calculateScore(Data feature,
                            boolean keepData,
                            float gain)
Calculates a score against the given feature. The score can be retreived with get score

Specified by:
calculateScore in interface Scoreable
Parameters:
feature - the feature to be scored
keepData - whether this Scoreable should keep a reference to the given feature
gain - gain to apply to the score;
Returns:
the score for the feature

normalizeScore

public float normalizeScore(float maxLogScore)
Normalizes a previously calculated score

Specified by:
normalizeScore in interface Scoreable
Parameters:
maxLogScore - the score to normalize this score with
Returns:
the normalized score

getWorkingScore

public float getWorkingScore()
Gets the working score. The working score is used to maintain non-final scores during the search. Some search algorithms such as bushderby use the working score

Returns:
the working score (in logMath log base)

setWorkingScore

public void setWorkingScore(float logScore)
Sets the working score for this token

Parameters:
logScore - the working score (in logMath log base)

setScore

public void setScore(float logScore)
Sets the score for this token

Parameters:
logScore - the new score for the token (in logMath log base)

getLanguageScore

public float getLanguageScore()
Returns the language score associated with this token

Returns:
the language score (in logMath log base)

getInsertionProbability

public float getInsertionProbability()
Returns the insertionPenalty associated with this token

Returns:
the insertion probability (in logMath log base)

getAcousticScore

public float getAcousticScore()
Returns the acoustic score for this token (in logMath log base)


getSearchState

public SearchState getSearchState()
Returns the SearchState associated with this token

Returns:
the searchState

isEmitting

public boolean isEmitting()
Determines if this token is associated with an emitting state. An emitting state is a state that can be scored acoustically.

Returns:
true if this token is associated with an emitting state

isFinal

public boolean isFinal()
Determines if this token is associated with a final SentenceHMM state.

Returns:
true if this token is associated with a final state

isWord

public boolean isWord()
Determines if this token marks the end of a word

Returns:
true if this token marks the end of a word

toString

public java.lang.String toString()
Retrieves the string representation of this object

Returns:
the string representation of this object

dumpTokenPath

public void dumpTokenPath()
dumps a branch of tokens


dumpTokenPath

public void dumpTokenPath(boolean includeHMMStates)
dumps a branch of tokens

Parameters:
includeHMMStates - if true include all sentence hmm states

getWordPath

public java.lang.String getWordPath(boolean wantFiller)
Returns the string of words leading up to this token.

Parameters:
wantFiller - if true, filler words are added
Returns:
the word path

getWordPathNoFiller

public java.lang.String getWordPathNoFiller()
Returns the string of words for this token, with no embedded filler words

Returns:
the string of words

getWordPath

public java.lang.String getWordPath()
Returns the string of words for this token, with embedded silences

Returns:
the string of words

getWordUnitPath

public java.lang.String getWordUnitPath()
Returns the string of words and units for this token, with embedded silences.

Returns:
the string of words and units

getWord

public Word getWord()
Returns the word of this Token, the search state is a WordSearchState. If the search state is not a WordSearchState, return null.

Returns:
the word of this Token, or null if this is not a word token

showCount

public static void showCount()
Shows the token count


getLocation

public final int getLocation()
Returns the location of this Token in the ActiveList. In the HeapActiveList implementation, it is the index of the Token in the array backing the heap.

Returns:
the location of this Token in the ActiveList

setLocation

public final void setLocation(int location)
Sets the location of this Token in the ActiveList.

Parameters:
location - the location of this Token

validate

public boolean validate()
Determines if this branch is valid

Returns:
true if the token and its predecessors are valid

getAppObject

public java.lang.Object getAppObject()
Returns the application object

Returns:
the application object

setAppObject

public void setAppObject(java.lang.Object obj)
Sets the application object

Parameters:
obj - the application object