Class NGramTokenizer

Tokenizes the input into n-grams of the given size(s).

On the contrary to NGramTokenFilter, this class sets offsets so that characters between startOffset and endOffset in the original stream are the same as the term chars.

For example, "abcde" would be tokenized as (minGram=2, maxGram=3):

TermPosition incrementPosition lengthOffsets
ab11[0,2[
abc11[0,3[
bc11[1,3[
bcd11[1,4[
cd11[2,4[
cde11[2,5[
de11[3,5[

This tokenizer changed a lot in Lucene 4.4 in order to:

tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
give the ability to pre-tokenize the stream (IsTokenChar(Int32)) before computing n-grams.

Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).

Although highly discouraged, it is still possible to use the old behavior through Lucene43NGramTokenizer.

Inheritance

System.Object

Lucene.Net.Util.AttributeSource

Lucene.Net.Analysis.TokenStream

Lucene.Net.Analysis.Tokenizer

NGramTokenizer

EdgeNGramTokenizer

Implements

System.IDisposable

Inherited Members

Lucene.Net.Analysis.Tokenizer.m_input

Tokenizer.Dispose(Boolean)

Tokenizer.CorrectOffset(Int32)

Tokenizer.SetReader(TextReader)

Lucene.Net.Analysis.TokenStream.Dispose()

Lucene.Net.Util.AttributeSource.GetAttributeFactory()

Lucene.Net.Util.AttributeSource.GetAttributeClassesEnumerator()

Lucene.Net.Util.AttributeSource.GetAttributeImplsEnumerator()

Lucene.Net.Util.AttributeSource.AddAttributeImpl(Lucene.Net.Util.Attribute)

Lucene.Net.Util.AttributeSource.AddAttribute<T>()

Lucene.Net.Util.AttributeSource.HasAttributes

Lucene.Net.Util.AttributeSource.HasAttribute<T>()

Lucene.Net.Util.AttributeSource.GetAttribute<T>()

Lucene.Net.Util.AttributeSource.ClearAttributes()

Lucene.Net.Util.AttributeSource.CaptureState()

Lucene.Net.Util.AttributeSource.RestoreState(Lucene.Net.Util.AttributeSource.State)

Lucene.Net.Util.AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

Lucene.Net.Util.AttributeSource.ReflectWith(Lucene.Net.Util.IAttributeReflector)

Lucene.Net.Util.AttributeSource.CloneAttributes()

Lucene.Net.Util.AttributeSource.CopyTo(Lucene.Net.Util.AttributeSource)

Lucene.Net.Util.AttributeSource.ToString()

System.Object.Equals(System.Object, System.Object)

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

Namespace: Lucene.Net.Analysis.NGram

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

public class NGramTokenizer : Tokenizer, IDisposable

Constructors

| Improve this Doc View Source

NGramTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader, Int32, Int32)

Creates NGramTokenizer with given min and max n-grams.

Declaration

public NGramTokenizer(LuceneVersion version, AttributeSource.AttributeFactory factory, TextReader input, int minGram, int maxGram)

Parameters

Type	Name	Description
Lucene.Net.Util.LuceneVersion	version	the lucene compatibility version
Lucene.Net.Util.AttributeSource.AttributeFactory	factory	Lucene.Net.Util.AttributeSource.AttributeFactory to use
System.IO.TextReader	input	System.IO.TextReader holding the input to be tokenized
System.Int32	minGram	the smallest n-gram to generate
System.Int32	maxGram	the largest n-gram to generate

| Improve this Doc View Source

NGramTokenizer(LuceneVersion, TextReader)

Creates NGramTokenizer with default min and max n-grams.

Declaration

public NGramTokenizer(LuceneVersion version, TextReader input)

Parameters

Type	Name	Description
Lucene.Net.Util.LuceneVersion	version	the lucene compatibility version
System.IO.TextReader	input	System.IO.TextReader holding the input to be tokenized

| Improve this Doc View Source

NGramTokenizer(LuceneVersion, TextReader, Int32, Int32)

Creates NGramTokenizer with given min and max n-grams.

Declaration

public NGramTokenizer(LuceneVersion version, TextReader input, int minGram, int maxGram)

Parameters

Type	Name	Description
Lucene.Net.Util.LuceneVersion	version	the lucene compatibility version
System.IO.TextReader	input	System.IO.TextReader holding the input to be tokenized
System.Int32	minGram	the smallest n-gram to generate
System.Int32	maxGram	the largest n-gram to generate

Fields

| Improve this Doc View Source

DEFAULT_MAX_NGRAM_SIZE

Declaration

public const int DEFAULT_MAX_NGRAM_SIZE = 2

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

DEFAULT_MIN_NGRAM_SIZE

Declaration

public const int DEFAULT_MIN_NGRAM_SIZE = 1

Field Value

Type	Description
System.Int32

Methods

| Improve this Doc View Source

End()

Declaration

public override sealed void End()

Overrides

Lucene.Net.Analysis.TokenStream.End()

| Improve this Doc View Source

IncrementToken()

Declaration

public override sealed bool IncrementToken()

Returns

Type	Description
System.Boolean

Overrides

Lucene.Net.Analysis.TokenStream.IncrementToken()

| Improve this Doc View Source

IsTokenChar(Int32)

Only collect characters which satisfy this condition.

Declaration

protected virtual bool IsTokenChar(int chr)

Parameters

Type	Name	Description
System.Int32	chr

Returns

Type	Description
System.Boolean

| Improve this Doc View Source

Reset()

Declaration

public override sealed void Reset()

Overrides

Lucene.Net.Analysis.Tokenizer.Reset()

Implements

System.IDisposable