Class ArabicLetterTokenizer

Tokenizer that breaks text into runs of letters and diacritics.

The problem with the standard Letter tokenizer is that it fails on diacritics. Handling similar to this is necessary for Indic Scripts, Hebrew, Thaana, etc.

You must specify the required Lucene.Net.Util.LuceneVersion compatibility when creating ArabicLetterTokenizer:

• As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See IsTokenChar(int) and Normalize(int) for details.

Inheritance

object

AttributeSource

TokenStream

Tokenizer

CharTokenizer

LetterTokenizer

ArabicLetterTokenizer

Implements

IDisposable

Inherited Members

CharTokenizer.Normalize(int)

CharTokenizer.IncrementToken()

CharTokenizer.End()

CharTokenizer.Reset()

Tokenizer.m_input

Tokenizer.Dispose(bool)

Tokenizer.CorrectOffset(int)

Tokenizer.SetReader(TextReader)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(object)

AttributeSource.ReflectAsString(bool)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

object.Equals(object, object)

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

Namespace: Lucene.Net.Analysis.Ar

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

[Obsolete("(3.1) Use StandardTokenizer instead.")]
public class ArabicLetterTokenizer : LetterTokenizer, IDisposable

Constructors

ArabicLetterTokenizer(LuceneVersion, AttributeFactory, TextReader)

Construct a new ArabicLetterTokenizer using a given Lucene.Net.Util.AttributeSource.AttributeFactory.

Declaration

public ArabicLetterTokenizer(LuceneVersion matchVersion, AttributeSource.AttributeFactory factory, TextReader @in)

Parameters

Type	Name	Description
LuceneVersion	matchVersion	Lucene version to match - See Lucene.Net.Util.LuceneVersion.
AttributeSource.AttributeFactory	factory	the attribute factory to use for this Tokenizer
TextReader	in	the input to split up into tokens

ArabicLetterTokenizer(LuceneVersion, TextReader)

Construct a new ArabicLetterTokenizer.

Declaration

public ArabicLetterTokenizer(LuceneVersion matchVersion, TextReader @in)

Parameters

Type	Name	Description
LuceneVersion	matchVersion	Lucene.Net.Util.LuceneVersion to match
TextReader	in	the input to split up into tokens

Methods

IsTokenChar(int)

Allows for Letter category or NonspacingMark category

Declaration

protected override bool IsTokenChar(int c)

Parameters

Type	Name	Description
int	c

Returns

Type	Description
bool

Overrides

LetterTokenizer.IsTokenChar(int)

Implements

IDisposable