A C E G I L M P R S T V

A

abbreviationList - Static variable in class com.parc.chat.tokenizer.MicroTextTokenizer

C

com.parc.chat.tokenizer - package com.parc.chat.tokenizer

E

EmoticonFST - Class in com.parc.chat.tokenizer

Finite-state transducer that recognizers emoticons.

EmoticonFST() - Constructor for class com.parc.chat.tokenizer.EmoticonFST

G

getCharacterPosition() - Method in class com.parc.chat.tokenizer.LabeledToken

Get character position where this token starts in the sentence.

getIndex() - Method in class com.parc.chat.tokenizer.LabeledToken

Get the index of this token into its sentence.

getNextState(LexicalFSA.State, LexicalFSA.Alphabet) - Static method in class com.parc.chat.tokenizer.LexicalFSA

getOriginalWord() - Method in class com.parc.chat.tokenizer.LabeledToken

Gets the token in its original form.

getStem() - Method in class com.parc.chat.tokenizer.LabeledToken

Gets the stemmed form (lemma) for this token.

getSymbol(char, char, char) - Static method in class com.parc.chat.tokenizer.LexicalFSA

getText() - Method in class com.parc.chat.tokenizer.MicroTextTokenizer

getTokenPosition(int) - Method in class com.parc.chat.tokenizer.MicroTextTokenizer

Retrieves the character position in the original string of the specified token.

getTokensAsList() - Method in class com.parc.chat.tokenizer.MicroTextTokenizer

Retrieves the list of tokens as strings.

getTokenType() - Method in class com.parc.chat.tokenizer.LabeledToken

Get the token type assigned to this token (see the TokenType enum).

I

isInitialEmoticonChar(char) - Static method in class com.parc.chat.tokenizer.EmoticonFST

L

LabeledToken - Class in com.parc.chat.tokenizer

Java bean class to hold a token that has been tagged with its lexical category and has been stemmed.

LabeledToken() - Constructor for class com.parc.chat.tokenizer.LabeledToken

Creates a new labeled token with none of its features set.

LabeledToken(String) - Constructor for class com.parc.chat.tokenizer.LabeledToken

Creates a new labeled token with its original word value set.

LexicalFSA - Class in com.parc.chat.tokenizer

LexicalFSA() - Constructor for class com.parc.chat.tokenizer.LexicalFSA

LexicalFSA.Alphabet - Enum in com.parc.chat.tokenizer

LexicalFSA.State - Enum in com.parc.chat.tokenizer

M

makeXML(Document) - Method in class com.parc.chat.tokenizer.LabeledToken

MicroTextTokenizer - Class in com.parc.chat.tokenizer

The MicroTextTokenizer class provides methods to break up a string into individual tokens.

MicroTextTokenizer(String) - Constructor for class com.parc.chat.tokenizer.MicroTextTokenizer

P

POS_ATTRIBUTE - Static variable in class com.parc.chat.tokenizer.LabeledToken

R

recognizeEmoticon(String) - Static method in class com.parc.chat.tokenizer.EmoticonFST

Determines if the next token in the supplied string is an emoticon.

S

setCharacterPosition(int) - Method in class com.parc.chat.tokenizer.LabeledToken

Set the character position where this token starts in the sentence.

setIndex(int) - Method in class com.parc.chat.tokenizer.LabeledToken

Set the index of this token into its sentence.

setOriginalWord(String) - Method in class com.parc.chat.tokenizer.LabeledToken

Sets the token.

setStem(String) - Method in class com.parc.chat.tokenizer.LabeledToken

Sets the stemmed form of this token.

setTokenType(TokenType) - Method in class com.parc.chat.tokenizer.LabeledToken

Set the token type on this token (see the TokenType enum).

STEM_ATTRIBUTE - Static variable in class com.parc.chat.tokenizer.LabeledToken

T

TOKEN_ELEMENT - Static variable in class com.parc.chat.tokenizer.LabeledToken

tokenize() - Method in class com.parc.chat.tokenizer.MicroTextTokenizer

Invokes the engine to execute the tokenizing process.

TokenType - Enum in com.parc.chat.tokenizer

This enum defines the types of tokens according to their use in text.

toString() - Method in class com.parc.chat.tokenizer.LabeledToken

Returns a string representation showing the original token and its part-of-speech separated by a slash.

toString() - Method in class com.parc.chat.tokenizer.MicroTextTokenizer

V

valueOf(String) - Static method in enum com.parc.chat.tokenizer.LexicalFSA.Alphabet

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.parc.chat.tokenizer.LexicalFSA.State

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum com.parc.chat.tokenizer.TokenType

Returns the enum constant of this type with the specified name.

values() - Static method in enum com.parc.chat.tokenizer.LexicalFSA.Alphabet

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.parc.chat.tokenizer.LexicalFSA.State

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum com.parc.chat.tokenizer.TokenType

Returns an array containing the constants of this enum type, in the order they are declared.

A C E G I L M P R S T V