PRECEDED_ROOT_OPTION, ROOT_TOKEN
Constructor and Description |
---|
DefaultTokenizer() |
Modifier and Type | Method and Description |
---|---|
TokenizationResult |
tokenize(java.lang.String input,
de.ims.icarus.util.Options options) |
public TokenizationResult tokenize(java.lang.String input, de.ims.icarus.util.Options options)
tokenize
in interface Tokenizer
Tokenizer.tokenize(java.lang.String, de.ims.icarus.util.Options)