Namespace Lucene.Net.Sandbox.Queries

Additional queries (some may have caveats or limitations)

Classes

DuplicateFilter

Filter to remove duplicate values from search results.

WARNING: for this to work correctly, you may have to wrap your reader as it cannot current deduplicate across different index segments.

Fuzzifies ALL terms provided as strings and then picks the best n differentiating terms. In effect this mixes the behaviour of Lucene.Net.Search.FuzzyQuery and MoreLikeThis but with special consideration of fuzzy scoring factors. This generally produces good results for queries where users may provide details in a number of fields and have no knowledge of boolean query syntax and also want a degree of fuzzy matching and a fast query.

For each source term the fuzzy variants are held in a Lucene.Net.Search.BooleanQuery with no coord factor (because we are not looking for matches on multiple variants in any one doc). Additionally, a specialized Lucene.Net.Search.TermQuery is used for variants and does not use that variant term's IDF because this would favour rarer terms eg misspellings. Instead, all variants use the same IDF ranking (the one for the source query term) and this is factored into the variant's boost. If the source query term does not exist in the index the average IDF of the variants is used.

SlowFuzzyQuery

Implements the classic fuzzy search query. The similarity measurement is based on the Levenshtein (edit distance) algorithm.

Note that, unlike Lucene.Net.Search.FuzzyQuery, this query will silently allow for a (possibly huge) number of edit distances in comparisons, and may be extremely slow (comparing every term in the index).

SlowFuzzyTermsEnum

Potentially slow fuzzy Lucene.Net.Index.TermsEnum for enumerating all terms that are similar to the specified filter term.

If the minSimilarity or maxEdits is greater than the Automaton's allowable range, this backs off to the classic (brute force) fuzzy terms enum method by calling GetAutomatonEnum(int, BytesRef).

Term enumerations are always ordered by Lucene.Net.Search.FuzzyTermsEnum.Comparer. Each term in the enumeration is greater than all that precede it.

SortedSetSortField

SortField for Lucene.Net.Index.SortedSetDocValues.

A Lucene.Net.Index.SortedSetDocValues contains multiple values for a field, so sorting with this technique "selects" a value as the representative sort value for the document.

By default, the minimum value in the set is selected as the sort value, but this can be customized. Selectors other than the default do have some limitations (see below) to ensure that all selections happen in constant-time for performance.

Like sorting by string, this also supports sorting missing values as first or last, via SetMissingValue(object).

Limitations:

Fields containing MaxValue or more unique values are unsupported.
Selectors other than the default MIN require optional codec support. However several codecs provided by Lucene, including the current default codec, support this.

Enums

KeepMode

KeepMode determines which document id to consider as the master, all others being identified as duplicates. Selecting the "first occurrence" can potentially save on IO.

ProcessingMode

"Full" processing mode starts by setting all bits to false and only setting bits for documents that contain the given field and are identified as none-duplicates.

"Fast" processing sets all bits to true then unsets all duplicate docs found for the given field. This approach avoids the need to read DocsEnum for terms that are seen to have a document frequency of exactly "1" (i.e. no duplicates). While a potentially faster approach , the downside is that bitsets produced will include bits set for documents that do not actually contain the field given.

Selector

Selects a value from the document's set to use as the sort value