A Tokenizer is a TokenStream whose input is a Reader.

This is an abstract class; subclasses must override {@link #IncrementToken()}

NOTE: Subclasses overriding {@link #next(Token)} must call {@link AttributeSource#ClearAttributes()} before setting attributes. Subclasses overriding {@link #IncrementToken()} must call {@link Token#Clear()} before setting Token attributes.

Namespace: Lucene.Net.Analysis
Assembly: Lucene.Net (in Lucene.Net.dll) Version: 2.9.4.1

Syntax

C#
public abstract class Tokenizer : TokenStream
Visual Basic
Public MustInherit Class Tokenizer _
	Inherits TokenStream
Visual C++
public ref class Tokenizer abstract : public TokenStream

Inheritance Hierarchy

See Also