Namespace Lucene.Net.Analysis.Snowball

TokenFilter and Analyzer implementations that use Snowball stemmers.

This project provides pre-compiled version of the Snowball stemmers based on revision 500 of the Tartarus Snowball repository, together with classes integrating them with the Lucene search engine.

A few changes has been made to the static Snowball code and compiled stemmers:

• Class SnowballProgram is made abstract and contains new abstract method stem() to avoid reflection in Lucene filter class SnowballFilter.

• All use of StringBuffers has been refactored to StringBuilder for speed.

• Snowball BSD license header has been added to the Java classes to avoid having RAT adding ASL headers.

  See the Snowball home page for more information about the algorithms.

  IMPORTANT NOTICE ON BACKWARDS COMPATIBILITY!

  An index created using the Snowball module in Lucene 2.3.2 and below might not be compatible with the Snowball module in Lucene 2.4 or greater.

  For more information about this issue see: https://issues.apache.org/jira/browse/LUCENE-1142

Classes

SnowballAnalyzer

Filters StandardTokenizer with StandardFilter, LowerCaseFilter, StopFilter and SnowballFilter.

Available stemmers are listed in org.tartarus.snowball.ext. The name of a stemmer is the part of the class name before "Stemmer", e.g., the stemmer in EnglishStemmer is named "English".

NOTE: This class uses the same LuceneVersion dependent settings as StandardAnalyzer, with the following addition:

• As of 3.1, uses TurkishLowerCaseFilter for Turkish language.

SnowballFilter

A filter that stems words using a Snowball-generated stemmer.

Available stemmers are listed in Lucene.Net.Tartarus.Snowball.Ext.

NOTE: SnowballFilter expects lowercased text.

• For the Turkish language, see TurkishLowerCaseFilter.
• For other languages, see LowerCaseFilter.

Note: This filter is aware of the KeywordAttribute. To prevent certain terms from being passed to the stemmer IsKeyword should be set to true in a previous TokenStream. Note: For including the original term as well as the stemmed version, see KeywordRepeatFilterFactory

SnowballPorterFilterFactory

Factory for SnowballFilter, with configurable language

Note: Use of the "Lovins" stemmer is not recommended, as it is implemented with reflection.

<fieldType name="text_snowballstem" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.SnowballPorterFilterFactory" protected="protectedkeyword.txt" language="English"/>
  </analyzer>
</fieldType>