Class Lucene43NGramTokenizer

Old broken version of NGramTokenizer.

Inheritance

object

AttributeSource

TokenStream

Tokenizer

Lucene43NGramTokenizer

Implements

IDisposable

Inherited Members

Tokenizer.SetReader(TextReader)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(object)

AttributeSource.ReflectAsString(bool)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

object.Equals(object, object)

object.GetType()

object.ReferenceEquals(object, object)

Namespace: Lucene.Net.Analysis.NGram

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

[Obsolete]
public sealed class Lucene43NGramTokenizer : Tokenizer, IDisposable

Constructors

Lucene43NGramTokenizer(AttributeFactory, TextReader, int, int)

Creates Lucene43NGramTokenizer with given min and max n-grams.

Declaration

public Lucene43NGramTokenizer(AttributeSource.AttributeFactory factory, TextReader input, int minGram, int maxGram)

Parameters

Type	Name	Description
AttributeSource.AttributeFactory	factory	Lucene.Net.Util.AttributeSource.AttributeFactory to use
TextReader	input	TextReader holding the input to be tokenized
int	minGram	the smallest n-gram to generate
int	maxGram	the largest n-gram to generate

Lucene43NGramTokenizer(TextReader)

Creates Lucene43NGramTokenizer with default min and max n-grams.

Declaration

public Lucene43NGramTokenizer(TextReader input)

Parameters

Type	Name	Description
TextReader	input	TextReader holding the input to be tokenized

Lucene43NGramTokenizer(TextReader, int, int)

Creates Lucene43NGramTokenizer with given min and max n-grams.

Declaration

public Lucene43NGramTokenizer(TextReader input, int minGram, int maxGram)

Parameters

Type	Name	Description
TextReader	input	TextReader holding the input to be tokenized
int	minGram	the smallest n-gram to generate
int	maxGram	the largest n-gram to generate

Fields

DEFAULT_MAX_NGRAM_SIZE

Old broken version of NGramTokenizer.

Declaration

public const int DEFAULT_MAX_NGRAM_SIZE = 2

Field Value

Type	Description
int

DEFAULT_MIN_NGRAM_SIZE

Old broken version of NGramTokenizer.

Declaration

public const int DEFAULT_MIN_NGRAM_SIZE = 1

Field Value

Type	Description
int

Methods

End()

This method is called by the consumer after the last token has been consumed, after Lucene.Net.Analysis.TokenStream.IncrementToken() returned false (using the new Lucene.Net.Analysis.TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.

If you override this method, always call base.End();.

Declaration

public override void End()

Overrides

Lucene.Net.Analysis.TokenStream.End()

Exceptions

Type	Condition
IOException	If an I/O error occurs

IncrementToken()

Returns the next token in the stream, or null at EOS.

Declaration

public override bool IncrementToken()

Returns

Type	Description
bool

Overrides

Lucene.Net.Analysis.TokenStream.IncrementToken()

Reset()

This method is called by a consumer before it begins consumption using Lucene.Net.Analysis.TokenStream.IncrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call base.Reset(), otherwise some internal state will not be correctly reset (e.g., Lucene.Net.Analysis.Tokenizer will throw InvalidOperationException on further usage).

Declaration

public override void Reset()

Overrides

Lucene.Net.Analysis.Tokenizer.Reset()

Implements

IDisposable