Class ICUTokenizerFactory

Factory for ICUTokenizer. Words are broken across script boundaries, then segmented according to the ICU4N.Text.BreakIterator and typing provided by the DefaultICUTokenizerConfig.

Inheritance

System.Object

Lucene.Net.Analysis.Util.AbstractAnalysisFactory

Lucene.Net.Analysis.Util.TokenizerFactory

ICUTokenizerFactory

Implements

Lucene.Net.Analysis.Util.IResourceLoaderAware

Inherited Members

Lucene.Net.Analysis.Util.TokenizerFactory.ForName(System.String, System.Collections.Generic.IDictionary<System.String, System.String>)

Lucene.Net.Analysis.Util.TokenizerFactory.LookupClass(System.String)

Lucene.Net.Analysis.Util.TokenizerFactory.AvailableTokenizers

Lucene.Net.Analysis.Util.TokenizerFactory.ReloadTokenizers()

Lucene.Net.Analysis.Util.TokenizerFactory.Create(System.IO.TextReader)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.LUCENE_MATCH_VERSION_PARAM

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.m_luceneMatchVersion

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.OriginalArgs

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.AssureMatchVersion()

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.LuceneMatchVersion

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.Require(System.Collections.Generic.IDictionary<System.String, System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.Require(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Collections.Generic.ICollection<System.String>)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.Require(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Collections.Generic.ICollection<System.String>, System.Boolean)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.Get(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.Get(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Collections.Generic.ICollection<System.String>)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.Get(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Collections.Generic.ICollection<System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.Get(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Collections.Generic.ICollection<System.String>, System.String, System.Boolean)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.RequireInt32(System.Collections.Generic.IDictionary<System.String, System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetInt32(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Int32)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.RequireBoolean(System.Collections.Generic.IDictionary<System.String, System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetBoolean(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Boolean)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.RequireSingle(System.Collections.Generic.IDictionary<System.String, System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetSingle(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Single)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.RequireChar(System.Collections.Generic.IDictionary<System.String, System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetChar(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Char)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetSet(System.Collections.Generic.IDictionary<System.String, System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetPattern(System.Collections.Generic.IDictionary<System.String, System.String>, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetCulture(System.Collections.Generic.IDictionary<System.String, System.String>, System.String, System.Globalization.CultureInfo)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetWordSet(Lucene.Net.Analysis.Util.IResourceLoader, System.String, System.Boolean)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetLines(Lucene.Net.Analysis.Util.IResourceLoader, System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetSnowballWordSet(Lucene.Net.Analysis.Util.IResourceLoader, System.String, System.Boolean)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.SplitFileNames(System.String)

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.GetClassArg()

Lucene.Net.Analysis.Util.AbstractAnalysisFactory.IsExplicitLuceneMatchVersion

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Namespace: Lucene.Net.Analysis.Icu.Segmentation

Assembly: Lucene.Net.ICU.dll

Syntax

public class ICUTokenizerFactory : TokenizerFactory, IResourceLoaderAware

Remarks

To use the default set of per-script rules:

<fieldType name="text_icu" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.ICUTokenizerFactory"/>
  </analyzer>
</fieldType>

You can customize this tokenizer's behavior by specifying per-script rule files, which are compiled by the ICU ICU4N.Text.RuleBasedBreakIterator. See the ICU RuleBasedBreakIterator syntax reference.

To add per-script rules, add a "rulefiles" argument, which should contain a comma-separated list of code:rulefile pairs in the following format: four-letter ISO 15924 script code, followed by a colon, then a resource path. E.g. to specify rules for Latin (script code "Latn") and Cyrillic (script code "Cyrl"):

<fieldType name="text_icu_custom" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.ICUTokenizerFactory" cjkAsWords="true"
               rulefiles="Latn:my.Latin.rules.rbbi,Cyrl:my.Cyrillic.rules.rbbi"/>
  </analyzer>
</fieldType>

Constructors

ICUTokenizerFactory(IDictionary<String, String>)

Creates a new ICUTokenizerFactory.

Declaration

public ICUTokenizerFactory(IDictionary<string, string> args)

Parameters

Type	Name	Description
System.Collections.Generic.IDictionary<System.String, System.String>	args

Methods

Create(AttributeSource.AttributeFactory, TextReader)

Declaration

public override Tokenizer Create(AttributeSource.AttributeFactory factory, TextReader input)

Parameters

Type	Name	Description
Lucene.Net.Util.AttributeSource.AttributeFactory	factory
System.IO.TextReader	input

Returns

Type	Description
Lucene.Net.Analysis.Tokenizer

Overrides

Lucene.Net.Analysis.Util.TokenizerFactory.Create(Lucene.Net.Util.AttributeSource.AttributeFactory, System.IO.TextReader)

Inform(IResourceLoader)

Declaration

public virtual void Inform(IResourceLoader loader)

Parameters

Type	Name	Description
Lucene.Net.Analysis.Util.IResourceLoader	loader

Implements

Lucene.Net.Analysis.Util.IResourceLoaderAware