Class SmartChineseAnalyzer

SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!

Note

This API is experimental and might change in incompatible ways in the next release.

Inheritance

object

Analyzer

SmartChineseAnalyzer

Implements

IDisposable

Inherited Members

Analyzer.NewAnonymous(Func<string, TextReader, TokenStreamComponents>)

Analyzer.NewAnonymous(Func<string, TextReader, TokenStreamComponents>, ReuseStrategy)

Analyzer.NewAnonymous(Func<string, TextReader, TokenStreamComponents>, Func<string, TextReader, TextReader>)

Analyzer.NewAnonymous(Func<string, TextReader, TokenStreamComponents>, Func<string, TextReader, TextReader>, ReuseStrategy)

Analyzer.GetTokenStream(string, TextReader)

Analyzer.GetTokenStream(string, string)

Analyzer.GetPositionIncrementGap(string)

Analyzer.GetOffsetGap(string)

Analyzer.Strategy

Analyzer.Dispose()

Analyzer.GLOBAL_REUSE_STRATEGY

Analyzer.PER_FIELD_REUSE_STRATEGY

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Lucene.Net.Analysis.Cn.Smart

Assembly: Lucene.Net.Analysis.SmartCn.dll

Syntax

public sealed class SmartChineseAnalyzer : Analyzer, IDisposable

Constructors

SmartChineseAnalyzer(LuceneVersion)

Create a new SmartChineseAnalyzer, using the default stopword list.

Declaration

public SmartChineseAnalyzer(LuceneVersion matchVersion)

Parameters

Type	Name	Description
LuceneVersion	matchVersion

SmartChineseAnalyzer(LuceneVersion, CharArraySet)

Create a new SmartChineseAnalyzer, using the provided Lucene.Net.Analysis.Util.CharArraySet of stopwords.

Note: the set should include punctuation, unless you want to index punctuation!

Declaration

public SmartChineseAnalyzer(LuceneVersion matchVersion, CharArraySet stopWords)

Parameters

Type	Name	Description
LuceneVersion	matchVersion
CharArraySet	stopWords	Lucene.Net.Analysis.Util.CharArraySet of stopwords to use.

SmartChineseAnalyzer(LuceneVersion, bool)

Create a new SmartChineseAnalyzer, optionally using the default stopword list.

The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

Declaration

public SmartChineseAnalyzer(LuceneVersion matchVersion, bool useDefaultStopWords)

Parameters

Type	Name	Description
LuceneVersion	matchVersion
bool	useDefaultStopWords	true to use the default stopword list.

Properties

DefaultStopSet

Returns an unmodifiable instance of the default stop-words set.

Declaration

public static CharArraySet DefaultStopSet { get; }

Property Value

Type	Description
CharArraySet	An unmodifiable instance of the default stop-words set.

Methods

CreateComponents(string, TextReader)

Creates a new Lucene.Net.Analysis.TokenStreamComponents instance for this analyzer.

Declaration

protected override TokenStreamComponents CreateComponents(string fieldName, TextReader reader)

Parameters

Type	Name	Description
string	fieldName	the name of the fields content passed to the Lucene.Net.Analysis.TokenStreamComponents sink as a reader
TextReader	reader	the reader passed to the Lucene.Net.Analysis.Tokenizer constructor

Returns

Type	Description
TokenStreamComponents	the Lucene.Net.Analysis.TokenStreamComponents for this analyzer.

Overrides

Analyzer.CreateComponents(string, TextReader)

Implements

IDisposable