Class CJKBigramFilter

Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

CJK types are set by these tokenizers, but you can also use CJKBigramFilter(TokenStream, CJKScript) to explicitly control which of the CJK scripts are turned into bigrams.

By default, when a CJK character has no adjacent characters to form a bigram, it is output in unigram form. If you want to always output both unigrams and bigrams, set the 

outputUnigrams
flag in CJKBigramFilter(TokenStream, CJKScript, Boolean). This can be used for a combined unigram+bigram approach.

In all cases, all non-CJK input is passed thru unmodified.

Inheritance
System.Object
AttributeSource
TokenStream
TokenFilter
CJKBigramFilter
Implements
IDisposable
Inherited Members
Namespace: Lucene.Net.Analysis.Cjk
Assembly: Lucene.Net.Analysis.Common.dll
Syntax
public sealed class CJKBigramFilter : TokenFilter, IDisposable

Constructors

| Improve this Doc View Source

CJKBigramFilter(TokenStream)

Calls CJKBigramFilter(TokenStream, CJKScript)

Declaration
public CJKBigramFilter(TokenStream in)
Parameters
Type Name Description
TokenStream in

Input TokenStream

| Improve this Doc View Source

CJKBigramFilter(TokenStream, CJKScript)

Calls CJKBigramFilter(TokenStream, CJKScript, Boolean)

Declaration
public CJKBigramFilter(TokenStream in, CJKScript flags)
Parameters
Type Name Description
TokenStream in

Input TokenStream

CJKScript flags

OR'ed set from HAN, HIRAGANA, KATAKANA, HANGUL

| Improve this Doc View Source

CJKBigramFilter(TokenStream, CJKScript, Boolean)

Create a new CJKBigramFilter, specifying which writing systems should be bigrammed, and whether or not unigrams should also be output.

Declaration
public CJKBigramFilter(TokenStream in, CJKScript flags, bool outputUnigrams)
Parameters
Type Name Description
TokenStream in

Input TokenStream

CJKScript flags

OR'ed set from HAN, HIRAGANA, KATAKANA, HANGUL

System.Boolean outputUnigrams

true if unigrams for the selected writing systems should also be output. when this is false, this is only done when there are no adjacent characters to form a bigram.

Fields

| Improve this Doc View Source

DOUBLE_TYPE

when we emit a bigram, its then marked as this type

Declaration
public const string DOUBLE_TYPE = null
Field Value
Type Description
System.String
| Improve this Doc View Source

SINGLE_TYPE

when we emit a unigram, its then marked as this type

Declaration
public const string SINGLE_TYPE = null
Field Value
Type Description
System.String

Methods

| Improve this Doc View Source

IncrementToken()

Declaration
public override bool IncrementToken()
Returns
Type Description
System.Boolean
Overrides
| Improve this Doc View Source

Reset()

Declaration
public override void Reset()
Overrides

Implements

IDisposable