Namespace Lucene.Net.Analysis.Synonym

Analysis components for Synonyms.

Classes

SolrSynonymParser

Parser for the Solr synonyms format.

  • Blank lines and lines starting with '#' are comments.
  • Explicit mappings match any token sequence on the LHS of "=>" and replace with all alternatives on the RHS. These types of mappings ignore the expand parameter in the constructor. Example:

    i-pod, i pod => ipod

  • Equivalent synonyms may be separated with commas and give no explicit mapping. In this case the mapping behavior will be taken from the expand parameter in the constructor. This allows the same synonym file to be used in different synonym handling strategies. Example:

    ipod, i-pod, i pod

  • Multiple synonym mapping entries are merged. Example:

        foo => foo bar
    foo => baz
    is equivalent to
    foo => foo bar, baz

Note

This API is experimental and might change in incompatible ways in the next release.

SynonymFilter

Matches single or multi word synonyms in a token stream. This token stream cannot properly handle position increments != 1, ie, you should place this filter before filtering out stop words.

Note that with the current implementation, parsing is greedy, so whenever multiple parses would apply, the rule starting the earliest and parsing the most tokens wins. For example if you have these rules: 

  a -> x
  a b -> y
  b c d -> z
Then input a b c d e parses to y b c d, ie the 2nd rule "wins" because it started earliest and matched the most input tokens of other rules starting at that point.

A future improvement to this filter could allow non-greedy parsing, such that the 3rd rule would win, and also separately allow multiple parses, such that all 3 rules would match, perhaps even on a rule by rule basis.

NOTE: when a match occurs, the output tokens associated with the matching rule are "stacked" on top of the input stream (if the rule had keepOrig=true) and also on top of another matched rule's output tokens. This is not a correct solution, as really the output should be an arbitrary graph/lattice. For example, with the above match, you would expect an exact Lucene.Net.Search.PhraseQuery "y b c" to match the parsed tokens, but it will fail to do so. This limitation is necessary because Lucene's Lucene.Net.Analysis.TokenStream (and index) cannot yet represent an arbitrary graph.

NOTE: If multiple incoming tokens arrive on the same position, only the first token at that position is used for parsing. Subsequent tokens simply pass through and are not parsed. A future improvement would be to allow these tokens to also be matched.

SynonymFilterFactory

Factory for SynonymFilter.

<fieldType name="text_synonym" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" 
            format="solr" ignoreCase="false" expand="true" 
            tokenizerFactory="solr.WhitespaceTokenizerFactory"
            [optional tokenizer factory parameters]/>
  </analyzer>
</fieldType>

An optional param name prefix of "tokenizerFactory." may be used for any init params that the SynonymFilterFactory needs to pass to the specified TokenizerFactory. If the TokenizerFactory expects an init parameters with the same name as an init param used by the SynonymFilterFactory, the prefix is mandatory.

The optional format parameter controls how the synonyms will be parsed: It supports the short names of solr for SolrSynonymParser and wordnet for and WordnetSynonymParser, or your own SynonymMap.Parser class name. The default is solr. A custom SynonymMap.Parser is expected to have a constructor taking:

  • System.Boolean dedup - true if duplicates should be ignored, false otherwise
  • System.Boolean expand - true if conflation groups should be expanded, false if they are one-directional
  • Lucene.Net.Analysis.Analyzer analyzer - an analyzer used for each raw synonym

SynonymMap

A map of synonyms, keys and values are phrases.

Note

This API is experimental and might change in incompatible ways in the next release.

SynonymMap.Builder

Builds an FSTSynonymMap.

Call Add(CharsRef, CharsRef, Boolean) until you have added all the mappings, then call Build() to get an FSTSynonymMap

Note

This API is experimental and might change in incompatible ways in the next release.

SynonymMap.Parser

Abstraction for parsing synonym files.

Note

This API is experimental and might change in incompatible ways in the next release.

WordnetSynonymParser

Parser for wordnet prolog format

See http://wordnet.princeton.edu/man/prologdb.5WN.html for a description of the format.

Note

This API is experimental and might change in incompatible ways in the next release.