Class Token

Analyzed token with morphological data from its dictionary.

Inheritance
System.Object
Token
Namespace: Lucene.Net.Analysis.Ja
Assembly: Lucene.Net.Analysis.Kuromoji.dll
Syntax
public class Token : object

Constructors

| Improve this Doc View Source

Token(Int32, Char[], Int32, Int32, JapaneseTokenizerType, Int32, IDictionary)

Declaration
public Token(int wordId, char[] surfaceForm, int offset, int length, JapaneseTokenizerType type, int position, IDictionary dictionary)
Parameters
Type Name Description
System.Int32 wordId
System.Char[] surfaceForm
System.Int32 offset
System.Int32 length
JapaneseTokenizerType type
System.Int32 position
IDictionary dictionary

Properties

| Improve this Doc View Source

Length

length of surfaceForm

Declaration
public virtual int Length { get; }
Property Value
Type Description
System.Int32
| Improve this Doc View Source

Offset

offset into surfaceForm

Declaration
public virtual int Offset { get; }
Property Value
Type Description
System.Int32
| Improve this Doc View Source

Position

Get index of this token in input text. Returns position of token.

Declaration
public virtual int Position { get; }
Property Value
Type Description
System.Int32
| Improve this Doc View Source

PositionLength

Gets or Sets the length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1.

Declaration
public virtual int PositionLength { get; set; }
Property Value
Type Description
System.Int32
| Improve this Doc View Source

SurfaceForm

surfaceForm

Declaration
public virtual char[] SurfaceForm { get; }
Property Value
Type Description
System.Char[]

Methods

| Improve this Doc View Source

GetBaseForm()

base form or null if token is not inflected

Declaration
public virtual string GetBaseForm()
Returns
Type Description
System.String

base form or null if token is not inflected
| Improve this Doc View Source

GetInflectionForm()

inflection form or null

Declaration
public virtual string GetInflectionForm()
Returns
Type Description
System.String

inflection form or null
| Improve this Doc View Source

GetInflectionType()

inflection type or null

Declaration
public virtual string GetInflectionType()
Returns
Type Description
System.String

inflection type or null
| Improve this Doc View Source

GetPartOfSpeech()

part of speech.

Declaration
public virtual string GetPartOfSpeech()
Returns
Type Description
System.String

part of speech.
| Improve this Doc View Source

GetPronunciation()

pronunciation. null if token doesn't have pronunciation.

Declaration
public virtual string GetPronunciation()
Returns
Type Description
System.String

pronunciation. null if token doesn't have pronunciation.
| Improve this Doc View Source

GetReading()

reading. null if token doesn't have reading.

Declaration
public virtual string GetReading()
Returns
Type Description
System.String

reading. null if token doesn't have reading.
| Improve this Doc View Source

GetSurfaceFormString()

surfaceForm as a String

Declaration
public virtual string GetSurfaceFormString()
Returns
Type Description
System.String

surfaceForm as a String
| Improve this Doc View Source

IsKnown()

Returns true if this token is known word.

Declaration
public virtual bool IsKnown()
Returns
Type Description
System.Boolean

true if this token is in standard dictionary. false if not.
| Improve this Doc View Source

IsUnknown()

Returns true if this token is unknown word.

Declaration
public virtual bool IsUnknown()
Returns
Type Description
System.Boolean

true if this token is unknown word. false if not.
| Improve this Doc View Source

IsUser()

Returns true if this token is defined in user dictionary.

Declaration
public virtual bool IsUser()
Returns
Type Description
System.Boolean

true if this token is in user dictionary. false if not.
| Improve this Doc View Source

ToString()

Declaration
public override string ToString()
Returns
Type Description
System.String