Class NGramTokenizer

Tokenizes the input into n-grams of the given size(s).

On the contrary to NGramTokenFilter, this class sets offsets so that characters between startOffset and endOffset in the original stream are the same as the term chars.

For example, "abcde" would be tokenized as (minGram=2, maxGram=3):

TermPosition incrementPosition lengthOffsets
ab11[0,2[
abc11[0,3[
bc11[1,3[
bcd11[1,4[
cd11[2,4[
cde11[2,5[
de11[3,5[

This tokenizer changed a lot in Lucene 4.4 in order to:

  • tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
  • count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
  • give the ability to pre-tokenize the stream (IsTokenChar(Int32)) before computing n-grams.

Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).

Although highly discouraged, it is still possible to use the old behavior through Lucene43NGramTokenizer.

Inheritance
System.Object
AttributeSource
TokenStream
Tokenizer
NGramTokenizer
EdgeNGramTokenizer
Implements
IDisposable
Inherited Members
Namespace: Lucene.Net.Analysis.NGram
Assembly: Lucene.Net.Analysis.Common.dll
Syntax
public class NGramTokenizer : Tokenizer, IDisposable

Constructors

| Improve this Doc View Source

NGramTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader, Int32, Int32)

Creates NGramTokenizer with given min and max n-grams.

Declaration
public NGramTokenizer(LuceneVersion version, AttributeSource.AttributeFactory factory, TextReader input, int minGram, int maxGram)
Parameters
Type Name Description
LuceneVersion version

the lucene compatibility version

AttributeSource.AttributeFactory factory

AttributeSource.AttributeFactory to use

TextReader input

holding the input to be tokenized

System.Int32 minGram

the smallest n-gram to generate

System.Int32 maxGram

the largest n-gram to generate

| Improve this Doc View Source

NGramTokenizer(LuceneVersion, TextReader)

Creates NGramTokenizer with default min and max n-grams.

Declaration
public NGramTokenizer(LuceneVersion version, TextReader input)
Parameters
Type Name Description
LuceneVersion version

the lucene compatibility version

TextReader input

holding the input to be tokenized

| Improve this Doc View Source

NGramTokenizer(LuceneVersion, TextReader, Int32, Int32)

Creates NGramTokenizer with given min and max n-grams.

Declaration
public NGramTokenizer(LuceneVersion version, TextReader input, int minGram, int maxGram)
Parameters
Type Name Description
LuceneVersion version

the lucene compatibility version

TextReader input

holding the input to be tokenized

System.Int32 minGram

the smallest n-gram to generate

System.Int32 maxGram

the largest n-gram to generate

Fields

| Improve this Doc View Source

DEFAULT_MAX_NGRAM_SIZE

Declaration
public const int DEFAULT_MAX_NGRAM_SIZE = null
Field Value
Type Description
System.Int32
| Improve this Doc View Source

DEFAULT_MIN_NGRAM_SIZE

Declaration
public const int DEFAULT_MIN_NGRAM_SIZE = null
Field Value
Type Description
System.Int32

Methods

| Improve this Doc View Source

End()

Declaration
public override sealed void End()
Overrides
| Improve this Doc View Source

IncrementToken()

Declaration
public override sealed bool IncrementToken()
Returns
Type Description
System.Boolean
Overrides
| Improve this Doc View Source

IsTokenChar(Int32)

Only collect characters which satisfy this condition.

Declaration
protected virtual bool IsTokenChar(int chr)
Parameters
Type Name Description
System.Int32 chr
Returns
Type Description
System.Boolean
| Improve this Doc View Source

Reset()

Declaration
public override sealed void Reset()
Overrides

Implements

IDisposable