Namespace Lucene.Net.Search.Highlight

The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages. The Highlighter class is the central component and can be used to extract the most interesting sections of a piece of text and highlight them, with the help of Fragmenter, fragment Scorer, and Formatter classes.

Example Usage

const LuceneVersion matchVersion = LuceneVersion.LUCENE_48;
Analyzer analyzer = new StandardAnalyzer(matchVersion);

// Create an index to search
string indexPath = Path.Combine(Path.GetTempPath(), Path.GetFileNameWithoutExtension(Path.GetTempFileName()));
Directory dir = FSDirectory.Open(indexPath);
using IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig(matchVersion, analyzer));

// This field must store term vectors and term vector offsets
var fieldType = new FieldType(TextField.TYPE_STORED)
{
    StoreTermVectors = true,
    StoreTermVectorOffsets = true
};
fieldType.Freeze();

// Create documents with two fields, one with term vectors (tv) and one without (notv)
writer.AddDocument(new Document {
    new Field("tv", "Thanks a million!", fieldType),
    new TextField("notv", "A million ways to win.", Field.Store.YES)
});
writer.AddDocument(new Document {
    new Field("tv", "Hopefully, this won't highlight a million times.", fieldType),
    new TextField("notv", "There are a million different ways to do that!", Field.Store.YES)
});

using IndexReader indexReader = writer.GetReader(applyAllDeletes: true);
writer.Dispose();

// Now search our index using an existing or new IndexReader

IndexSearcher searcher = new IndexSearcher(indexReader);
QueryParser parser = new QueryParser(matchVersion, "notv", analyzer);
Query query = parser.Parse("million");

TopDocs hits = searcher.Search(query, 10);

SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter();
Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query));
int totalScoreDocs = hits.ScoreDocs.Length > 10 ? 10 : hits.ScoreDocs.Length;
for (int i = 0; i < totalScoreDocs; i++)
{
    int id = hits.ScoreDocs[i].Doc;
    Document doc = searcher.Doc(id);
    string text = doc.Get("notv");
    TokenStream tokenStream = TokenSources.GetAnyTokenStream(searcher.IndexReader, id, "notv", analyzer);
    TextFragment[] frag = highlighter.GetBestTextFragments(
        tokenStream, text, mergeContiguousFragments: false, maxNumFragments: 10); // highlighter.GetBestFragments(tokenStream, text, 3, "...");
    for (int j = 0; j < frag.Length; j++)
    {
        if (frag[j] != null && frag[j].Score > 0)
        {
            Console.WriteLine(frag[j].ToString());
        }
    }
    //Term vector
    text = doc.Get("tv");
    tokenStream = TokenSources.GetAnyTokenStream(searcher.IndexReader, hits.ScoreDocs[i].Doc, "tv", analyzer);
    frag = highlighter.GetBestTextFragments(tokenStream, text, false, 10);
    for (int j = 0; j < frag.Length; j++)
    {
        if (frag[j] != null && frag[j].Score > 0)
        {
            Console.WriteLine(frag[j].ToString());
        }
    }
    Console.WriteLine("-------------");
}

New features 2005-02-06

This release adds options for encoding (thanks to Nicko Cadell). An "Encoder" implementation such as the new SimpleHTMLEncoder class can be passed to the highlighter to encode all those non-xhtml standard characters such as & into legal values. This simple class may not suffice for some languages - Commons Lang has an implementation that could be used: escapeHtml(String) in http://svn.apache.org/viewcvs.cgi/jakarta/commons/proper/lang/trunk/src/java/org/apache/commons/lang/StringEscapeUtils.java?rev=137958&view=markup

New features 2004-12-22

This release adds some new capabilities:

Faster highlighting using Term vector support
New formatting options to use color intensity to show informational value
Options for better summarization by using term IDF scores to influence fragment selection

The highlighter takes a <xref:Lucene.Net.Analysis.TokenStream> as input. Until now these streams have typically been produced using an <xref:Lucene.Net.Analysis.Analyzer> but the new class TokenSources provides helper methods for obtaining TokenStreams from the new TermVector position support (see latest CVS version).

The new class GradientFormatter can use a scale of colors to highlight terms according to their score. A subtle use of color can help emphasize the reasons for matching (useful when doing "MoreLikeThis" queries and you want to see what the basis of the similarities are).

The QueryScorer class has a new constructor which can use an <xref:Lucene.Net.Index.IndexReader> to derive the IDF (inverse document frequency) for each term in order to influence the score. This is useful for helping to extracting the most significant sections of a document and in supplying scores used by the new GradientFormatter to color significant words more strongly. The QueryScorer.MaxTermWeight method is useful when passed to the GradientFormatter constructor to define the top score which is associated with the top color.

Classes

DefaultEncoder

Simple IEncoder implementation that does not modify the output

GradientFormatter

Formats text with different color intensity depending on the score of the term.

Highlighter

Class used to markup highlighted terms found in the best sections of a text, using configurable IFragmenter, Lucene.Net.Search.Scorer, IFormatter, IEncoder and tokenizers.

InvalidTokenOffsetsException

Exception thrown if TokenStream Tokens are incompatible with provided text

NullFragmenter

IFragmenter implementation which does not fragment the text. This is useful for highlighting the entire content of a document or field.

OffsetLimitTokenFilter

This Lucene.Net.Analysis.TokenFilter limits the number of tokens while indexing by adding up the current offset.

PositionSpan

Utility class to record Positions Spans

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

QueryScorer

IScorer implementation which scores text fragments by the number of unique query terms found. This class converts appropriate Lucene.Net.Search.Querys to Lucene.Net.Search.Spans.SpanQuerys and attempts to score only those terms that participated in generating the 'hit' on the document.

QueryTermExtractor

Utility class used to extract the terms used in a query, plus any weights. This class will not find terms for Lucene.Net.Search.MultiTermQuery, RangeQuery and Lucene.Net.Search.PrefixQuery classes so the caller must pass a rewritten query (see Lucene.Net.Search.Query.Rewrite(Lucene.Net.Index.IndexReader)) to obtain a list of expanded terms.

QueryTermScorer

IScorer implementation which scores text fragments by the number of unique query terms found. This class uses the QueryTermExtractor class to process determine the query terms and their boosts to be used.

SimpleFragmenter

IFragmenter implementation which breaks text up into same-size fragments with no concerns over spotting sentence boundaries.

SimpleHTMLEncoder

Simple IEncoder implementation to escape text for HTML output

SimpleHTMLFormatter

Simple IFormatter implementation to highlight terms with a pre and post tag

SimpleSpanFragmenter

IFragmenter implementation which breaks text up into same-size fragments but does not split up Lucene.Net.Search.Spans.Spans. This is a simple sample class.

SpanGradientFormatter

Formats text with different color intensity depending on the score of the term using the span tag. GradientFormatter uses a bgcolor argument to the font tag which doesn't work in Mozilla, thus this class.

TextFragment

Low-level class used to record information about a section of a document with a score.

TokenGroup

One, or several overlapping tokens, along with the score(s) and the scope of the original text

TokenSources

Hides implementation issues associated with obtaining a Lucene.Net.Analysis.TokenStream for use with the Highlighter - can obtain from term vectors with offsets and positions or from an Analyzer re-parsing the stored content. see TokenStreamFromTermVector

TokenStreamFromTermPositionVector

Lucene.Net.Analysis.TokenStream created from a term vector field.

WeightedSpanTerm

Lightweight class to hold term, weight, and positions used for scoring this term.

WeightedSpanTermExtractor

Class used to extract WeightedSpanTerms from a Lucene.Net.Search.Query based on whether Lucene.Net.Index.Terms from the Lucene.Net.Search.Query are contained in a supplied Lucene.Net.Analysis.TokenStream.

WeightedSpanTermExtractor.PositionCheckingMap<K>

This class makes sure that if both position sensitive and insensitive versions of the same term are added, the position insensitive one wins.

WeightedTerm

Lightweight class to hold term and a Weight value used for scoring this term