Class LetterTokenizer

A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined by IsLetter(int) predicate.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

You must specify the required Lucene.Net.Util.LuceneVersion compatibility when creating LetterTokenizer:

• As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See IsTokenChar(int) and Normalize(int) for details.

Inheritance

object

AttributeSource

TokenStream

Tokenizer

CharTokenizer

LetterTokenizer

ArabicLetterTokenizer

LowerCaseTokenizer

Implements

IDisposable

Inherited Members

CharTokenizer.Normalize(int)

CharTokenizer.IncrementToken()

CharTokenizer.End()

CharTokenizer.Reset()

Tokenizer.m_input

Tokenizer.Dispose(bool)

Tokenizer.CorrectOffset(int)

Tokenizer.SetReader(TextReader)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(object)

AttributeSource.ReflectAsString(bool)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

object.Equals(object, object)

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Namespace: Lucene.Net.Analysis.Core

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

public class LetterTokenizer : CharTokenizer, IDisposable

Constructors

LetterTokenizer(LuceneVersion, AttributeFactory, TextReader)

Construct a new LetterTokenizer using a given Lucene.Net.Util.AttributeSource.AttributeFactory.

Declaration

public LetterTokenizer(LuceneVersion matchVersion, AttributeSource.AttributeFactory factory, TextReader @in)

Parameters

Type	Name	Description
LuceneVersion	matchVersion	Lucene.Net.Util.LuceneVersion to match
AttributeSource.AttributeFactory	factory	the attribute factory to use for this Lucene.Net.Analysis.Tokenizer
TextReader	in	the input to split up into tokens

LetterTokenizer(LuceneVersion, TextReader)

Construct a new LetterTokenizer.

Declaration

public LetterTokenizer(LuceneVersion matchVersion, TextReader @in)

Parameters

Type	Name	Description
LuceneVersion	matchVersion	Lucene.Net.Util.LuceneVersion to match.
TextReader	in	the input to split up into tokens

Methods

IsTokenChar(int)

Collects only characters which satisfy IsLetter(int).

Declaration

protected override bool IsTokenChar(int c)

Parameters

Type	Name	Description
int	c

Returns

Type	Description
bool

Overrides

CharTokenizer.IsTokenChar(int)

Implements

IDisposable