com.github.chen0040.data.text

Class BasicTokenizer

java.lang.Object

com.github.chen0040.data.text.BasicTokenizer

All Implemented Interfaces:

Tokenizer, Serializable

public class BasicTokenizer
extends Object
implements Tokenizer, Serializable

Created by xschen on 9/10/15.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
BasicTokenizer()

Method Summary

Modifier and Type	Method and Description
static List<String>	doTokenize(List<String> text)
static List<String>	doTokenize(String text)
List<String>	tokenize(String str) Tokenizes a string using the algorithms by Grefenstette (1999) and Palmer (2000).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BasicTokenizer

public BasicTokenizer()

Method Detail

tokenize

public List<String> tokenize(String str)

Tokenizes a string using the algorithms by Grefenstette (1999) and Palmer (2000).

Specified by:

tokenize in interface Tokenizer

doTokenize

public static List<String> doTokenize(String text)

doTokenize

public static List<String> doTokenize(List<String> text)