Class ICUTokenizer

Breaks text into words according to UAX #29: Unicode Text Segmentation (http://www.unicode.org/reports/tr29/)

Words are broken across script boundaries, then segmented according to the BreakIterator and typing provided by the ICUTokenizerConfig

This is a Lucene.NET EXPERIMENTAL API, use at your own risk
Inheritance
System.Object
AttributeSource
TokenStream
Tokenizer
ICUTokenizer
Implements
IDisposable
Inherited Members
Namespace: Lucene.Net.Analysis.Icu.Segmentation
Assembly: Lucene.Net.ICU.dll
Syntax
public sealed class ICUTokenizer : Tokenizer, IDisposable

Constructors

| Improve this Doc View Source

ICUTokenizer(AttributeSource.AttributeFactory, TextReader, ICUTokenizerConfig)

Construct a new ICUTokenizer that breaks text into words from the given , using a tailored configuration.

Declaration
public ICUTokenizer(AttributeSource.AttributeFactory factory, TextReader input, ICUTokenizerConfig config)
Parameters
Type Name Description
AttributeSource.AttributeFactory factory

AttributeSource.AttributeFactory to use.
TextReader input

containing text to tokenize.
ICUTokenizerConfig config

Tailored configuration.
| Improve this Doc View Source

ICUTokenizer(TextReader)

Construct a new ICUTokenizer that breaks text into words from the given .

Declaration
public ICUTokenizer(TextReader input)
Parameters
Type Name Description
TextReader input

containing text to tokenize.
Remarks

The default script-specific handling is used.

The default attribute factory is used.

See Also
| Improve this Doc View Source

ICUTokenizer(TextReader, ICUTokenizerConfig)

Construct a new ICUTokenizer that breaks text into words from the given , using a tailored configuration.

Declaration
public ICUTokenizer(TextReader input, ICUTokenizerConfig config)
Parameters
Type Name Description
TextReader input

containing text to tokenize.
ICUTokenizerConfig config

Tailored configuration.
Remarks

The default attribute factory is used.

Methods

| Improve this Doc View Source

End()

Declaration
public override void End()
Overrides
| Improve this Doc View Source

IncrementToken()

Declaration
public override bool IncrementToken()
Returns
Type Description
System.Boolean
Overrides
| Improve this Doc View Source

Reset()

Declaration
public override void Reset()
Overrides

Implements

IDisposable

See Also