The StandardTokenizer type exposes the following members.

Fields

  NameDescription
Public fieldStatic memberACRONYM
Public fieldStatic memberACRONYM_DEP Obsolete.
Public fieldStatic memberALPHANUM
Public fieldStatic memberAPOSTROPHE
Public fieldStatic memberCJ
Public fieldStatic memberCOMPANY
Public fieldStatic memberEMAIL
Public fieldStatic memberHOST
Protected fieldinput
The text source for this Tokenizer.
(Inherited from Tokenizer.)
Public fieldStatic memberNUM
Public fieldStatic memberTOKEN_TYPES
String token types that correspond to token type int constants
Public fieldStatic membertokenImage Obsolete.

See Also