Class ICUTokenizer

Breaks text into words according to UAX #29: Unicode Text Segmentation (http://www.unicode.org/reports/tr29/)

Words are broken across script boundaries, then segmented according to the BreakIterator and typing provided by the ICUTokenizerConfig

Note

This API is experimental and might change in incompatible ways in the next release.

Inheritance
object
AttributeSource
TokenStream
Tokenizer
ICUTokenizer
Implements
Inherited Members
TokenStream.Dispose()
AttributeSource.GetAttributeFactory()
AttributeSource.GetAttributeClassesEnumerator()
AttributeSource.GetAttributeImplsEnumerator()
AttributeSource.AddAttributeImpl(Attribute)
AttributeSource.AddAttribute<T>()
AttributeSource.HasAttributes
AttributeSource.HasAttribute<T>()
AttributeSource.GetAttribute<T>()
AttributeSource.ClearAttributes()
AttributeSource.CaptureState()
AttributeSource.RestoreState(AttributeSource.State)
AttributeSource.GetHashCode()
AttributeSource.ReflectWith(IAttributeReflector)
AttributeSource.CloneAttributes()
AttributeSource.CopyTo(AttributeSource)
AttributeSource.ToString()
Namespace: Lucene.Net.Analysis.Icu.Segmentation
Assembly: Lucene.Net.ICU.dll
Syntax
public sealed class ICUTokenizer : Tokenizer, IDisposable

Constructors

ICUTokenizer(AttributeFactory, TextReader, ICUTokenizerConfig)

Construct a new ICUTokenizer that breaks text into words from the given TextReader, using a tailored ICU4N.Text.BreakIterator configuration.

Declaration
public ICUTokenizer(AttributeSource.AttributeFactory factory, TextReader input, ICUTokenizerConfig config)
Parameters
Type Name Description
AttributeSource.AttributeFactory factory

Lucene.Net.Util.AttributeSource.AttributeFactory to use.
TextReader input

TextReader containing text to tokenize.
ICUTokenizerConfig config

Tailored ICU4N.Text.BreakIterator configuration.
See Also

ICUTokenizer(TextReader)

Construct a new ICUTokenizer that breaks text into words from the given TextReader.

Declaration
public ICUTokenizer(TextReader input)
Parameters
Type Name Description
TextReader input

TextReader containing text to tokenize.
Remarks

The default script-specific handling is used.

The default attribute factory is used.
See Also

ICUTokenizer(TextReader, ICUTokenizerConfig)

Construct a new ICUTokenizer that breaks text into words from the given TextReader, using a tailored ICU4N.Text.BreakIterator configuration.

Declaration
public ICUTokenizer(TextReader input, ICUTokenizerConfig config)
Parameters
Type Name Description
TextReader input

TextReader containing text to tokenize.
ICUTokenizerConfig config

Tailored ICU4N.Text.BreakIterator configuration.
Remarks

The default attribute factory is used.

See Also

Methods

End()

This method is called by the consumer after the last token has been consumed, after Lucene.Net.Analysis.TokenStream.IncrementToken() returned false (using the new Lucene.Net.Analysis.TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.

If you override this method, always call base.End();.
Declaration
public override void End()
Overrides
Lucene.Net.Analysis.TokenStream.End()
Exceptions
Type Condition
IOException

If an I/O error occurs
See Also

IncrementToken()

Consumers (i.e., Lucene.Net.Index.IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate Lucene.Net.Util.IAttributes with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use Lucene.Net.Util.AttributeSource.CaptureState() to create a copy of the current attribute state.

this method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to Lucene.Net.Util.AttributeSource.AddAttribute<T>() and Lucene.Net.Util.AttributeSource.GetAttribute<T>(), references to all Lucene.Net.Util.IAttributes that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in Lucene.Net.Analysis.TokenStream.IncrementToken().
Declaration
public override bool IncrementToken()
Returns
Type Description
bool

false for end of stream; true otherwise
Overrides
Lucene.Net.Analysis.TokenStream.IncrementToken()
See Also

Reset()

This method is called by a consumer before it begins consumption using Lucene.Net.Analysis.TokenStream.IncrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call base.Reset(), otherwise some internal state will not be correctly reset (e.g., Lucene.Net.Analysis.Tokenizer will throw InvalidOperationException on further usage).
Declaration
public override void Reset()
Overrides
Lucene.Net.Analysis.Tokenizer.Reset()
See Also

Implements

See Also