Namespace Lucene.Net.Analysis.Util

Utility functions for text analysis.

Classes

AbstractAnalysisFactory

Abstract parent class for analysis factories TokenizerFactory, TokenFilterFactory and CharFilterFactory.

The typical lifecycle for a factory consumer is:

  • Create factory via its constructor (or via XXXFactory.ForName)
  • (Optional) If the factory uses resources such as files, Inform(IResourceLoader) is called to initialize those resources.
  • Consumer calls create() to obtain instances.

BufferedCharFilter

LUCENENET specific class to mimic Java's BufferedReader (that is, a reader that is seekable) so it supports Mark() and Reset() (which are part of the Java Reader class), but also provide the Correct() method of BaseCharFilter.

CharArrayDictionary

Utility functions for text analysis.

CharArrayDictionary<TValue>

A simple class that stores text strings as char[]'s in a hash table. Note that this is not a general purpose class. For example, it cannot remove items from the dictionary, nor does it resize its hash table to be smaller, etc. It is designed to be quick to retrieve items by char[] keys without the necessity of converting to a string first.

You must specify the required Lucene.Net.Util.LuceneVersion compatibility when creating CharArrayDictionary<TValue>:

  • As of 3.1, supplementary characters are properly lowercased.
Before 3.1 supplementary characters could not be lowercased correctly due to the lack of Unicode 4 support in JDK 1.4. To use instances of CharArrayDictionary<TValue> with the behavior before Lucene 3.1 pass a Lucene.Net.Util.LuceneVersion < 3.1 to the constructors.

CharArrayDictionary<TValue>.Enumerator

Enumerates the elements of a CharArrayDictionary<TValue>.

This enumerator exposes CurrentKey efficient access to the underlying char[]. It also has CurrentKeyString, CurrentKeyCharSequence, and CurrentValue properties for convenience.

CharArrayDictionary<TValue>.ValueCollection

Class that represents the values in the CharArrayDictionary<TValue>.

CharArraySet

A simple class that stores strings as char[]'s in a hash table. Note that this is not a general purpose class. For example, it cannot remove items from the set, nor does it resize its hash table to be smaller, etc. It is designed to be quick to test if a char[] is in the set without the necessity of converting it to a string first.

You must specify the required Lucene.Net.Util.LuceneVersion compatibility when creating CharArraySet:

  • As of 3.1, supplementary characters are properly lowercased.
Before 3.1 supplementary characters could not be lowercased correctly due to the lack of Unicode 4 support in JDK 1.4. To use instances of CharArraySet with the behavior before Lucene 3.1 pass a Lucene.Net.Util.LuceneVersion to the constructors.

Please note: This class implements ISet<T> but does not behave like it should in all cases. The generic type is string, because you can add any object to it, that has a string representation (which is converted to a string). The add methods will use ToString() and store the result using a char[] buffer. The same behavior have the Contains(string) methods. The GetEnumerator() returns an IEnumerator{char[]}

CharFilterFactory

Abstract parent class for analysis factories that create Lucene.Net.Analysis.CharFilter instances.

CharTokenizer

An abstract base class for simple, character-oriented tokenizers.

You must specify the required Lucene.Net.Util.LuceneVersion compatibility when creating CharTokenizer:

A new CharTokenizer API has been introduced with Lucene 3.1. This API moved from UTF-16 code units to UTF-32 codepoints to eventually add support for supplementary characters. The old char based API has been deprecated and should be replaced with the int based methods IsTokenChar(int) and Normalize(int).

As of Lucene 3.1 each CharTokenizer - constructor expects a Lucene.Net.Util.LuceneVersion argument. Based on the given Lucene.Net.Util.LuceneVersion either the new API or a backwards compatibility layer is used at runtime. For Lucene.Net.Util.LuceneVersion < 3.1 the backwards compatibility layer ensures correct behavior even for indexes build with previous versions of Lucene. If a Lucene.Net.Util.LuceneVersion >= 3.1 is used CharTokenizer requires the new API to be implemented by the instantiated class. Yet, the old char based API is not required anymore even if backwards compatibility must be preserved. CharTokenizer subclasses implementing the new API are fully backwards compatible if instantiated with Lucene.Net.Util.LuceneVersion < 3.1.

Note: If you use a subclass of CharTokenizer with Lucene.Net.Util.LuceneVersion >= 3.1 on an index build with a version < 3.1, created tokens might not be compatible with the terms in your index.

CharacterUtils

CharacterUtils provides a unified interface to Character-related operations to implement backwards compatible character operations based on a Lucene.Net.Util.LuceneVersion instance.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

CharacterUtils.CharacterBuffer

A simple IO buffer to use with Fill(CharacterBuffer, TextReader).

ClasspathResourceLoader

Simple IResourceLoader that uses GetManifestResourceStream(string) and GetType(string) to open resources and Types, respectively.

DictionaryExtensions

Extensions to IDictionary<TKey, TValue> for CharArrayDictionary<TValue>.

ElisionFilter

Removes elisions from a Lucene.Net.Analysis.TokenStream. For example, "l'avion" (the plane) will be tokenized as "avion" (plane).

Elision in Wikipedia

ElisionFilterFactory

Factory for ElisionFilter.

<fieldType name="text_elsn" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ElisionFilterFactory" 
      articles="stopwordarticles.txt" ignoreCase="true"/>
  </analyzer>
</fieldType>

EnumerableExtensions

Extensions to IEnumerable<T> for CharArraySet.

FilesystemResourceLoader

Simple IResourceLoader that opens resource files from the local file system, optionally resolving against a base directory.

This loader wraps a delegate IResourceLoader that is used to resolve all files, the current base directory does not contain. NewInstance<T>(string) is always resolved against the delegate, as an System.Assembly is needed.

You can chain several FilesystemResourceLoaders to allow lookup of files in more than one base directory.

FilteringTokenFilter

Abstract base class for TokenFilters that may remove tokens. You have to implement Accept() and return a boolean if the current token should be preserved. IncrementToken() uses this method to decide if a token should be passed to the caller.

As of Lucene 4.4, an ArgumentException is thrown when trying to disable position increments when filtering terms.

OpenStringBuilder

A StringBuilder that allows one to access the array.

RollingCharBuffer

Acts like a forever growing char[] as you read characters into it from the provided reader, but internally it uses a circular buffer to only hold the characters that haven't been freed yet. This is like a PushbackReader, except you don't have to specify up-front the max size of the buffer, but you do have to periodically call FreeBefore(int).

StemmerUtil

Some commonly-used stemming functions

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

StopwordAnalyzerBase

Base class for Lucene.Net.Analysis.Analyzers that need to make use of stopword sets.

TokenFilterFactory

Abstract parent class for analysis factories that create Lucene.Net.Analysis.TokenFilter instances.

TokenizerFactory

Abstract parent class for analysis factories that create Lucene.Net.Analysis.Tokenizer instances.

WordlistLoader

Loader for text files that represent a list of stopwords.

Lucene.Net.Util.IOUtils to obtain TextReader instances.
Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Structs

CharArrayDictionary<TValue>.ValueCollection.Enumerator

Enumerates the elements of a CharArrayDictionary<TValue>.ValueCollection.

CharArraySet.Enumerator

Enumerates the elements of a CharArraySet object.

This implementation provides direct access to the char[] array of the underlying collection as well as convenience properties for converting to string and J2N.Text.ICharSequence.

Interfaces

IMultiTermAwareComponent

Add to any analysis factory component to allow returning an analysis component factory for use with partial terms in prefix queries, wildcard queries, range query endpoints, regex queries, etc.

Note

This API is experimental and might change in incompatible ways in the next release.

IResourceLoader

Abstraction for loading resources (streams, files, and classes).

IResourceLoaderAware

Interface for a component that needs to be initialized by an implementation of IResourceLoader.