Class ShingleFilter

A ShingleFilter constructs shingles (token n-grams) from a token stream. In other words, it creates combinations of tokens as a single token.

For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".

This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_"). It does not handle a position increment of 0.

Inheritance

System.Object

AttributeSource

TokenStream

TokenFilter

ShingleFilter

Implements

IDisposable

Inherited Members

TokenFilter.m_input

TokenFilter.Dispose(Boolean)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

Namespace: Lucene.Net.Analysis.Shingle

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

public sealed class ShingleFilter : TokenFilter, IDisposable

Constructors

| Improve this Doc View Source

ShingleFilter(TokenStream)

Construct a ShingleFilter with default shingle size: 2.

Declaration

public ShingleFilter(TokenStream input)

Parameters

Type	Name	Description
TokenStream	input	input stream

| Improve this Doc View Source

ShingleFilter(TokenStream, Int32)

Constructs a ShingleFilter with the specified shingle size from the TokenStream input

Declaration

public ShingleFilter(TokenStream input, int maxShingleSize)

Parameters

Type	Name	Description
TokenStream	input	input stream
System.Int32	maxShingleSize	maximum shingle size produced by the filter.

| Improve this Doc View Source

ShingleFilter(TokenStream, Int32, Int32)

Constructs a ShingleFilter with the specified shingle size from the TokenStream input

Declaration

public ShingleFilter(TokenStream input, int minShingleSize, int maxShingleSize)

Parameters

Type	Name	Description
TokenStream	input	input stream
System.Int32	minShingleSize	minimum shingle size produced by the filter.
System.Int32	maxShingleSize	maximum shingle size produced by the filter.

| Improve this Doc View Source

ShingleFilter(TokenStream, String)

Construct a ShingleFilter with the specified token type for shingle tokens and the default shingle size: 2

Declaration

public ShingleFilter(TokenStream input, string tokenType)

Parameters

Type	Name	Description
TokenStream	input	input stream
System.String	tokenType	token type for shingle tokens

Fields

| Improve this Doc View Source

DEFAULT_FILLER_TOKEN

filler token for when positionIncrement is more than 1

Declaration

public const string DEFAULT_FILLER_TOKEN = null

Field Value

Type	Description
System.String

| Improve this Doc View Source

DEFAULT_MAX_SHINGLE_SIZE

default maximum shingle size is 2.

Declaration

public const int DEFAULT_MAX_SHINGLE_SIZE = null

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

DEFAULT_MIN_SHINGLE_SIZE

default minimum shingle size is 2.

Declaration

public const int DEFAULT_MIN_SHINGLE_SIZE = null

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

DEFAULT_TOKEN_SEPARATOR

The default string to use when joining adjacent tokens to form a shingle

Declaration

public const string DEFAULT_TOKEN_SEPARATOR = null

Field Value

Type	Description
System.String

| Improve this Doc View Source

DEFAULT_TOKEN_TYPE

default token type attribute value is "shingle"

Declaration

public const string DEFAULT_TOKEN_TYPE = null

Field Value

Type	Description
System.String

Methods

| Improve this Doc View Source

End()

Declaration

public override void End()

Overrides

TokenFilter.End()

| Improve this Doc View Source

IncrementToken()

Declaration

public override bool IncrementToken()

Returns

Type	Description
System.Boolean

Overrides

TokenStream.IncrementToken()

| Improve this Doc View Source

Reset()

Declaration

public override void Reset()

Overrides

TokenFilter.Reset()

| Improve this Doc View Source

SetFillerToken(String)

Sets the string to insert for each position at which there is no token (i.e., when position increment is greater than one).

Declaration

public void SetFillerToken(string fillerToken)

Parameters

Type	Name	Description
System.String	fillerToken	string to insert at each position where there is no token

| Improve this Doc View Source

SetMaxShingleSize(Int32)

Set the max shingle size (default: 2)

Declaration

public void SetMaxShingleSize(int maxShingleSize)

Parameters

Type	Name	Description
System.Int32	maxShingleSize	max size of output shingles

| Improve this Doc View Source

SetMinShingleSize(Int32)

Set the min shingle size (default: 2).

This method requires that the passed in minShingleSize is not greater than maxShingleSize, so make sure that maxShingleSize is set before calling this method.

The unigram output option is independent of the min shingle size.

Declaration

public void SetMinShingleSize(int minShingleSize)

Parameters

Type	Name	Description
System.Int32	minShingleSize	min size of output shingles

| Improve this Doc View Source

SetOutputUnigrams(Boolean)

Shall the output stream contain the input tokens (unigrams) as well as shingles? (default: true.)

Declaration

public void SetOutputUnigrams(bool outputUnigrams)

Parameters

Type	Name	Description
System.Boolean	outputUnigrams	Whether or not the output stream shall contain the input tokens (unigrams)

| Improve this Doc View Source

SetOutputUnigramsIfNoShingles(Boolean)

Shall we override the behavior of outputUnigrams==false for those times when no shingles are available (because there are fewer than minShingleSize tokens in the input stream)? (default: false.)

Note that if outputUnigrams==true, then unigrams are always output, regardless of whether any shingles are available.

Declaration

public void SetOutputUnigramsIfNoShingles(bool outputUnigramsIfNoShingles)

Parameters

Type	Name	Description
System.Boolean	outputUnigramsIfNoShingles	Whether or not to output a single unigram when no shingles are available.

| Improve this Doc View Source

SetTokenSeparator(String)

Sets the string to use when joining adjacent tokens to form a shingle

Declaration

public void SetTokenSeparator(string tokenSeparator)

Parameters

Type	Name	Description
System.String	tokenSeparator	used to separate input stream tokens in output shingles

| Improve this Doc View Source

SetTokenType(String)

Set the type of the shingle tokens produced by this filter. (default: "shingle")

Declaration

public void SetTokenType(string tokenType)

Parameters

Type	Name	Description
System.String	tokenType	token tokenType

Implements

IDisposable