Class LetterTokenizer

A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined by J2N.Character.IsLetter(System.Int32) predicate.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

You must specify the required Lucene.Net.Util.LuceneVersion compatibility when creating LetterTokenizer:

As of 3.1, CharTokenizer uses an System.Int32 based API to normalize and detect token characters. See IsTokenChar(Int32) and Normalize(Int32) for details.

Inheritance

System.Object

Lucene.Net.Util.AttributeSource

Lucene.Net.Analysis.TokenStream

Lucene.Net.Analysis.Tokenizer

CharTokenizer

LetterTokenizer

ArabicLetterTokenizer

LowerCaseTokenizer

Implements

System.IDisposable

Inherited Members

CharTokenizer.Normalize(Int32)

CharTokenizer.IncrementToken()

CharTokenizer.End()

CharTokenizer.Reset()

Lucene.Net.Analysis.Tokenizer.m_input

Tokenizer.Dispose(Boolean)

Tokenizer.CorrectOffset(Int32)

Tokenizer.SetReader(TextReader)

Lucene.Net.Analysis.TokenStream.Dispose()

Lucene.Net.Util.AttributeSource.GetAttributeFactory()

Lucene.Net.Util.AttributeSource.GetAttributeClassesEnumerator()

Lucene.Net.Util.AttributeSource.GetAttributeImplsEnumerator()

Lucene.Net.Util.AttributeSource.AddAttributeImpl(Lucene.Net.Util.Attribute)

Lucene.Net.Util.AttributeSource.AddAttribute<T>()

Lucene.Net.Util.AttributeSource.HasAttributes

Lucene.Net.Util.AttributeSource.HasAttribute<T>()

Lucene.Net.Util.AttributeSource.GetAttribute<T>()

Lucene.Net.Util.AttributeSource.ClearAttributes()

Lucene.Net.Util.AttributeSource.CaptureState()

Lucene.Net.Util.AttributeSource.RestoreState(Lucene.Net.Util.AttributeSource.State)

Lucene.Net.Util.AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

Lucene.Net.Util.AttributeSource.ReflectWith(Lucene.Net.Util.IAttributeReflector)

Lucene.Net.Util.AttributeSource.CloneAttributes()

Lucene.Net.Util.AttributeSource.CopyTo(Lucene.Net.Util.AttributeSource)

Lucene.Net.Util.AttributeSource.ToString()

System.Object.Equals(System.Object, System.Object)

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

Namespace: Lucene.Net.Analysis.Core

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

public class LetterTokenizer : CharTokenizer, IDisposable

Constructors

| Improve this Doc View Source

LetterTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader)

Construct a new LetterTokenizer using a given Lucene.Net.Util.AttributeSource.AttributeFactory.

Declaration

public LetterTokenizer(LuceneVersion matchVersion, AttributeSource.AttributeFactory factory, TextReader in)

Parameters

Type	Name	Description
Lucene.Net.Util.LuceneVersion	matchVersion	Lucene.Net.Util.LuceneVersion to match
Lucene.Net.Util.AttributeSource.AttributeFactory	factory	the attribute factory to use for this Lucene.Net.Analysis.Tokenizer
System.IO.TextReader	in	the input to split up into tokens

| Improve this Doc View Source

LetterTokenizer(LuceneVersion, TextReader)

Construct a new LetterTokenizer.

Declaration

public LetterTokenizer(LuceneVersion matchVersion, TextReader in)

Parameters

Type	Name	Description
Lucene.Net.Util.LuceneVersion	matchVersion	Lucene.Net.Util.LuceneVersion to match.
System.IO.TextReader	in	the input to split up into tokens

Methods

| Improve this Doc View Source

IsTokenChar(Int32)

Collects only characters which satisfy J2N.Character.IsLetter(System.Int32).

Declaration

protected override bool IsTokenChar(int c)

Parameters

Type	Name	Description
System.Int32	c

Returns

Type	Description
System.Boolean

Overrides

CharTokenizer.IsTokenChar(Int32)

Implements

System.IDisposable