Package com.parc.chat.tokenizer

Class Summary

Class	Description
EmoticonFST	Finite-state transducer that recognizers emoticons.
LabeledToken	Java bean class to hold a token that has been tagged with its lexical category and has been stemmed.
LexicalFSA
MicroTextTokenizer	The MicroTextTokenizer class provides methods to break up a string into individual tokens.

Enum Summary

Enum	Description
LexicalFSA.Alphabet
LexicalFSA.State
TokenType	This enum defines the types of tokens according to their use in text.