com.parc.chat.tokenizer

Class MicroTextTokenizer

java.lang.Object

com.parc.chat.tokenizer.MicroTextTokenizer

public class MicroTextTokenizer
extends Object

The MicroTextTokenizer class provides methods to break up a string into individual tokens. It has special features to deal with micro-text as it appears in micro-blogs like Twitter and SMS and chat messages. Obtain a new instance for each string to be tokenized. Typical usage:

 
 MicroTextTokenizer tokenizer = new MicroTextTokenizer("It's easier to buy something than to make something.");
 Stack<LabeledToken> tokenStack = tokenizer.tokenize();
 LabeledToken firstToken = tokenStack.get(0);
 ...
 
 

Author:

Kyle Dent

Field Summary

Fields

Modifier and Type	Field and Description
static List<String>	abbreviationList

Constructor Summary

Constructors

Constructor and Description
MicroTextTokenizer(String tweetString)

Method Summary

Modifier and Type	Method and Description
String	getText()
int	getTokenPosition(int tokenIndex) Retrieves the character position in the original string of the specified token.
List<String>	getTokensAsList() Retrieves the list of tokens as strings.
Stack<LabeledToken>	tokenize() Invokes the engine to execute the tokenizing process.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

abbreviationList

public static List<String> abbreviationList

Constructor Detail

MicroTextTokenizer

public MicroTextTokenizer(String tweetString)

Method Detail

tokenize

public Stack<LabeledToken> tokenize()

Invokes the engine to execute the tokenizing process. This method must be called before other methods that provide token information. It breaks the original text string into words and symbols, and assigns attributes to each one creating a LabeledToken object for each token.

Returns:

a Stack of LabeledTokens

getText

public String getText()

toString

public String toString()

Overrides:

toString in class Object

getTokensAsList

public List<String> getTokensAsList()

Retrieves the list of tokens as strings. The list contains the tokens in their surface form as they appeared in the original string. You must call tokenize() before calling this method or an IllegalStateException will be thrown.

Returns:

a list of tokenized Strings

getTokenPosition

public int getTokenPosition(int tokenIndex)

Retrieves the character position in the original string of the specified token.

Parameters:

tokenIndex - the index into the token stack

Returns:

an integer giving the character position of the token