Lucene.Net  3.0.3
Lucene.Net is a port of the Lucene search engine library, written in C# and targeted at .NET runtime users.
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Properties Pages
Classes | Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Properties | List of all members
Lucene.Net.Search.Similar.MoreLikeThis Class Reference

Generate "more like this" similarity queries. Based on this mail: More...

Public Member Functions

 MoreLikeThis (IndexReader ir)
 Constructor requiring an IndexReader.
 
 MoreLikeThis (IndexReader ir, Lucene.Net.Search.Similarity sim)
 
void SetMaxDocFreqPct (int maxPercentage)
 Set the maximum percentage in which words may still appear. Words that appear in more than this many percent of all docs will be ignored.
 
System.String[] GetFieldNames ()
 Returns the field names that will be used when generating the 'More Like This' query. The default field names that will be used is DEFAULT_FIELD_NAMES.
 
void SetFieldNames (System.String[] fieldNames)
 Sets the field names that will be used when generating the 'More Like This' query. Set this to null for the field names to be determined at runtime from the IndexReader provided in the constructor.
 
void SetStopWords (ISet< string > stopWords)
 Set the set of stopwords. Any word in this set is considered "uninteresting" and ignored. Even if your Analyzer allows stopwords, you might want to tell the MoreLikeThis code to ignore them, as for the purposes of document similarity it seems reasonable to assume that "a stop word is never interesting".
 
ISet< string > GetStopWords ()
 Get the current stop words being used.
 
Query Like (int docNum)
 Return a query that will return docs like the passed lucene document ID.
 
Query Like (System.IO.FileInfo f)
 Return a query that will return docs like the passed file.
 
Query Like (System.Uri u)
 Return a query that will return docs like the passed URL.
 
Query Like (System.IO.Stream is_Renamed)
 Return a query that will return docs like the passed stream.
 
Query Like (System.IO.TextReader r)
 Return a query that will return docs like the passed Reader.
 
System.String DescribeParams ()
 Describe the parameters that control how the "more like this" query is formed.
 
PriorityQueue< object[]> RetrieveTerms (System.IO.TextReader r)
 Find words for a more-like-this query former. The result is a priority queue of arrays with one entry for every word in the document. Each array has 6 elements. The elements are:
 
System.String[] RetrieveInterestingTerms (int docNum)
 
System.String[] RetrieveInterestingTerms (System.IO.TextReader r)
 Convenience routine to make it easy to return the most interesting words in a document. More advanced users will call RetrieveTerms(System.IO.TextReader) directly.
 

Static Public Member Functions

static void Main (System.String[] a)
 Test driver. Pass in "-i INDEX" and then either "-fn FILE" or "-url URL".
 

Public Attributes

const int DEFAULT_MAX_NUM_TOKENS_PARSED = 5000
 Default maximum number of tokens to parse in each example doc field that is not stored with TermVector support.
 
const int DEFAULT_MIN_TERM_FREQ = 2
 Ignore terms with less than this frequency in the source doc.
 
const int DEFAULT_MIN_DOC_FREQ = 5
 Ignore words which do not occur in at least this many docs.
 
const int DEFAULT_MAX_DOC_FREQ = int.MaxValue
 Ignore words wich occur in more than this many docs
 
const bool DEFAULT_BOOST = false
 Boost terms in query based on score.
 
const int DEFAULT_MIN_WORD_LENGTH = 0
 Ignore words less than this length or if 0 then this has no effect.
 
const int DEFAULT_MAX_WORD_LENGTH = 0
 Ignore words greater than this length or if 0 then this has no effect.
 
const int DEFAULT_MAX_QUERY_TERMS = 25
 Return a Query with no more than this many terms.
 

Static Public Attributes

static readonly Analyzer DEFAULT_ANALYZER = new StandardAnalyzer(Util.Version.LUCENE_CURRENT)
 Default analyzer to parse source doc with.
 
static readonly System.String[] DEFAULT_FIELD_NAMES = new System.String[] { "contents" }
 Default field names. Null is used to specify that the field names should be looked up at runtime from the provided reader.
 
static readonly ISet< string > DEFAULT_STOP_WORDS = null
 Default set of stopwords. If null means to allow stop words.
 

Properties

float BoostFactor [get, set]
 Gets or sets the boost factor used when boosting terms
 
Similarity Similarity [get, set]
 
Analyzer Analyzer [get, set]
 Gets or sets the analyzer used to parse source doc with. The default analyzer is the DEFAULT_ANALYZER.
 
int MinTermFreq [get, set]
 Gets or sets the frequency below which terms will be ignored in the source doc. The default frequency is the DEFAULT_MIN_TERM_FREQ.
 
int MinDocFreq [get, set]
 Gets or sets the frequency at which words will be ignored which do not occur in at least this many docs. The default frequency is DEFAULT_MIN_DOC_FREQ.
 
int MaxDocFreq [get, set]
 Gets or sets the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored. The default frequency is DEFAULT_MAX_DOC_FREQ
 
bool Boost [get, set]
 Gets or sets a boolean indicating whether to boost terms in query based on "score" or not. The default is DEFAULT_BOOST.
 
int MinWordLen [get, set]
 Gets or sets the minimum word length below which words will be ignored. Set this to 0 for no minimum word length. The default is DEFAULT_MIN_WORD_LENGTH.
 
int MaxWordLen [get, set]
 Gets or sets the maximum word length above which words will be ignored. Set this to 0 for no maximum word length. The default is DEFAULT_MAX_WORD_LENGTH.
 
int MaxQueryTerms [get, set]
 Gets or sets the maximum number of query terms that will be included in any generated query. The default is DEFAULT_MAX_QUERY_TERMS.
 
int MaxNumTokensParsed [get, set]
 Gets or sets the maximum number of tokens to parse in each example doc field that is not stored with TermVector support
 

Detailed Description

Generate "more like this" similarity queries. Based on this mail:

Lucene does let you access the document frequency of terms, with IndexReader.DocFreq().
Term frequencies can be computed by re-tokenizing the text, which, for a single document,
is usually fast enough.  But looking up the DocFreq() of every term in the document is
probably too slow.
You can use some heuristics to prune the set of terms, to avoid calling DocFreq() too much,
or at all.  Since you're trying to maximize a tf*idf score, you're probably most interested
in terms with a high tf. Choosing a tf threshold even as low as two or three will radically
reduce the number of terms under consideration.  Another heuristic is that terms with a
high idf (i.e., a low df) tend to be longer.  So you could threshold the terms by the
number of characters, not selecting anything less than, e.g., six or seven characters.
With these sorts of heuristics you can usually find small set of, e.g., ten or fewer terms
that do a pretty good job of characterizing a document.
It all depends on what you're trying to do.  If you're trying to eek out that last percent
of precision and recall regardless of computational difficulty so that you can win a TREC
competition, then the techniques I mention above are useless.  But if you're trying to
provide a "more like this" button on a search results page that does a decent job and has
good performance, such techniques might be useful.
An efficient, effective "more-like-this" query generator would be a great contribution, if
anyone's interested.  I'd imagine that it would take a Reader or a String (the document's
text), analyzer Analyzer, and return a set of representative terms using heuristics like those
above.  The frequency and length thresholds could be parameters, etc.
Doug

Initial Usage

This class has lots of options to try to make it efficient and flexible. See the body of Main below in the source for real code, or if you want pseudo code, the simpliest possible usage is as follows. The bold fragment is specific to this class.

IndexReader ir = ...
IndexSearcher is = ...

MoreLikeThis mlt = new MoreLikeThis(ir);
Reader target = ... // orig source of doc you want to find similarities to
Query query = mlt.Like( target);

Hits hits = is.Search(query);
// now the usual iteration thru 'hits' - the only thing to watch for is to make sure
you ignore the doc if it matches your 'target' document, as it should be similar to itself 

Thus you:

  1. do your normal, Lucene setup for searching,
  2. create a MoreLikeThis,
  3. get the text of the doc you want to find similaries to
  4. then call one of the Like() calls to generate a similarity query
  5. call the searcher to find the similar docs

More Advanced Usage

You may want to use SetFieldNames so you can examine multiple fields (e.g. body and title) for similarity.

Depending on the size of your index and the size and makeup of your documents you may want to call the other set methods to control how the similarity queries are generated:


Changes: Mark Harwood 29/02/04
Some bugfixing, some refactoring, some optimisation.

  • bugfix: retrieveTerms(int docNum) was not working for indexes without a termvector -added missing code
  • bugfix: No significant terms being created for fields with a termvector - because was only counting one occurence per term/field pair in calculations(ie not including frequency info from TermVector)
  • refactor: moved common code into isNoiseWord()
  • optimise: when no termvector support available - used maxNumTermsParsed to limit amount of tokenization
  

Definition at line 138 of file MoreLikeThis.cs.

Constructor & Destructor Documentation

Lucene.Net.Search.Similar.MoreLikeThis.MoreLikeThis ( IndexReader  ir)

Constructor requiring an IndexReader.

Definition at line 274 of file MoreLikeThis.cs.

Lucene.Net.Search.Similar.MoreLikeThis.MoreLikeThis ( IndexReader  ir,
Lucene.Net.Search.Similarity  sim 
)

Definition at line 278 of file MoreLikeThis.cs.

Member Function Documentation

System.String Lucene.Net.Search.Similar.MoreLikeThis.DescribeParams ( )

Describe the parameters that control how the "more like this" query is formed.

Definition at line 613 of file MoreLikeThis.cs.

System.String [] Lucene.Net.Search.Similar.MoreLikeThis.GetFieldNames ( )

Returns the field names that will be used when generating the 'More Like This' query. The default field names that will be used is DEFAULT_FIELD_NAMES.

Returns
the field names that will be used when generating the 'More Like This' query.

Definition at line 364 of file MoreLikeThis.cs.

ISet<string> Lucene.Net.Search.Similar.MoreLikeThis.GetStopWords ( )

Get the current stop words being used.

See Also
SetStopWords

Definition at line 423 of file MoreLikeThis.cs.

Query Lucene.Net.Search.Similar.MoreLikeThis.Like ( int  docNum)

Return a query that will return docs like the passed lucene document ID.

Parameters
docNumthe documentID of the lucene doc to generate the 'More Like This" query for.
Returns
a query that will return docs like the passed lucene document ID.

Definition at line 453 of file MoreLikeThis.cs.

Query Lucene.Net.Search.Similar.MoreLikeThis.Like ( System.IO.FileInfo  f)

Return a query that will return docs like the passed file.

Returns
a query that will return docs like the passed file.

Definition at line 470 of file MoreLikeThis.cs.

Query Lucene.Net.Search.Similar.MoreLikeThis.Like ( System.Uri  u)

Return a query that will return docs like the passed URL.

Returns
a query that will return docs like the passed URL.

Definition at line 487 of file MoreLikeThis.cs.

Query Lucene.Net.Search.Similar.MoreLikeThis.Like ( System.IO.Stream  is_Renamed)

Return a query that will return docs like the passed stream.

Returns
a query that will return docs like the passed stream.

Definition at line 497 of file MoreLikeThis.cs.

Query Lucene.Net.Search.Similar.MoreLikeThis.Like ( System.IO.TextReader  r)

Return a query that will return docs like the passed Reader.

Returns
a query that will return docs like the passed Reader.

Definition at line 507 of file MoreLikeThis.cs.

static void Lucene.Net.Search.Similar.MoreLikeThis.Main ( System.String[]  a)
static

Test driver. Pass in "-i INDEX" and then either "-fn FILE" or "-url URL".

Definition at line 638 of file MoreLikeThis.cs.

System.String [] Lucene.Net.Search.Similar.MoreLikeThis.RetrieveInterestingTerms ( int  docNum)

Definition at line 872 of file MoreLikeThis.cs.

System.String [] Lucene.Net.Search.Similar.MoreLikeThis.RetrieveInterestingTerms ( System.IO.TextReader  r)

Convenience routine to make it easy to return the most interesting words in a document. More advanced users will call RetrieveTerms(System.IO.TextReader) directly.

Parameters
rthe source document
Returns

the most interesting words in the document

See Also
RetrieveTerms(System.IO.TextReader)
See Also
MaxQueryTerms

Definition at line 901 of file MoreLikeThis.cs.

PriorityQueue<object[]> Lucene.Net.Search.Similar.MoreLikeThis.RetrieveTerms ( System.IO.TextReader  r)

Find words for a more-like-this query former. The result is a priority queue of arrays with one entry for every word in the document. Each array has 6 elements. The elements are:

  1. The word (String)
  2. The top field that this word comes from (String)
  3. The score for this word (Float)
  4. The IDF value (Float)
  5. The frequency of this word in the index (Integer)
  6. The frequency of this word in the source document (Integer)

This is a somewhat "advanced" routine, and in general only the 1st entry in the array is of interest. This method is exposed so that you can identify the "interesting words" in a document. For an easier method to call see RetrieveInterestingTerms(System.IO.TextReader).

Parameters
rthe reader that has the content of the document
Returns

the most intresting words in the document ordered by score, with the highest scoring, or best entry, first

See Also
RetrieveInterestingTerms(System.IO.TextReader)

Definition at line 860 of file MoreLikeThis.cs.

void Lucene.Net.Search.Similar.MoreLikeThis.SetFieldNames ( System.String[]  fieldNames)

Sets the field names that will be used when generating the 'More Like This' query. Set this to null for the field names to be determined at runtime from the IndexReader provided in the constructor.

Parameters
fieldNamesthe field names that will be used when generating the 'More Like This' query.

Definition at line 377 of file MoreLikeThis.cs.

void Lucene.Net.Search.Similar.MoreLikeThis.SetMaxDocFreqPct ( int  maxPercentage)

Set the maximum percentage in which words may still appear. Words that appear in more than this many percent of all docs will be ignored.

Parameters
maxPercentagethe maximum percentage of documents (0-100) that a term may appear in to be still considered relevant

Definition at line 344 of file MoreLikeThis.cs.

void Lucene.Net.Search.Similar.MoreLikeThis.SetStopWords ( ISet< string >  stopWords)

Set the set of stopwords. Any word in this set is considered "uninteresting" and ignored. Even if your Analyzer allows stopwords, you might want to tell the MoreLikeThis code to ignore them, as for the purposes of document similarity it seems reasonable to assume that "a stop word is never interesting".

Parameters
stopWordsset of stopwords, if null it means to allow stop words
See Also
Lucene.Net.Analysis.StopFilter.MakeStopSet(string[])
See Also
GetStopWords

Definition at line 415 of file MoreLikeThis.cs.

Member Data Documentation

readonly Analyzer Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_ANALYZER = new StandardAnalyzer(Util.Version.LUCENE_CURRENT)
static

Default analyzer to parse source doc with.

See Also
Analyzer

Definition at line 150 of file MoreLikeThis.cs.

const bool Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_BOOST = false

Boost terms in query based on score.

See Also
Boost
See Also
Boost

Definition at line 178 of file MoreLikeThis.cs.

readonly System.String [] Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_FIELD_NAMES = new System.String[] { "contents" }
static

Default field names. Null is used to specify that the field names should be looked up at runtime from the provided reader.

Definition at line 183 of file MoreLikeThis.cs.

const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_DOC_FREQ = int.MaxValue

Ignore words wich occur in more than this many docs

See Also
MaxDocFreq, MaxDocFreq

Definition at line 171 of file MoreLikeThis.cs.

const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_NUM_TOKENS_PARSED = 5000

Default maximum number of tokens to parse in each example doc field that is not stored with TermVector support.

See Also
MaxNumTokensParsed

Definition at line 144 of file MoreLikeThis.cs.

const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_QUERY_TERMS = 25

Return a Query with no more than this many terms.

See Also
BooleanQuery.MaxClauseCount
See Also
MaxQueryTerms
See Also
MaxQueryTerms

Definition at line 221 of file MoreLikeThis.cs.

const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_WORD_LENGTH = 0

Ignore words greater than this length or if 0 then this has no effect.

See Also
MaxWordLen
See Also
MaxWordLen

Definition at line 197 of file MoreLikeThis.cs.

const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MIN_DOC_FREQ = 5

Ignore words which do not occur in at least this many docs.

See Also
MinDocFreq
See Also
MinDocFreq

Definition at line 164 of file MoreLikeThis.cs.

const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MIN_TERM_FREQ = 2

Ignore terms with less than this frequency in the source doc.

See Also
MinTermFreq
See Also
MinTermFreq

Definition at line 157 of file MoreLikeThis.cs.

const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MIN_WORD_LENGTH = 0

Ignore words less than this length or if 0 then this has no effect.

See Also
MinWordLen
See Also
MinWordLen

Definition at line 190 of file MoreLikeThis.cs.

readonly ISet<string> Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_STOP_WORDS = null
static

Default set of stopwords. If null means to allow stop words.

See Also
SetStopWords
See Also
GetStopWords

Definition at line 207 of file MoreLikeThis.cs.

Property Documentation

Analyzer Lucene.Net.Search.Similar.MoreLikeThis.Analyzer
getset

Gets or sets the analyzer used to parse source doc with. The default analyzer is the DEFAULT_ANALYZER.

An analyzer is not required for generating a query with the Like(int) method, all other 'like' methods require an analyzer.

the analyzer that will be used to parse source doc with.

See Also
DEFAULT_ANALYZER

Definition at line 300 of file MoreLikeThis.cs.

bool Lucene.Net.Search.Similar.MoreLikeThis.Boost
getset

Gets or sets a boolean indicating whether to boost terms in query based on "score" or not. The default is DEFAULT_BOOST.

Definition at line 353 of file MoreLikeThis.cs.

float Lucene.Net.Search.Similar.MoreLikeThis.BoostFactor
getset

Gets or sets the boost factor used when boosting terms

Definition at line 268 of file MoreLikeThis.cs.

int Lucene.Net.Search.Similar.MoreLikeThis.MaxDocFreq
getset

Gets or sets the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored. The default frequency is DEFAULT_MAX_DOC_FREQ

Definition at line 331 of file MoreLikeThis.cs.

int Lucene.Net.Search.Similar.MoreLikeThis.MaxNumTokensParsed
getset

Gets or sets the maximum number of tokens to parse in each example doc field that is not stored with TermVector support

See Also
DEFAULT_MAX_NUM_TOKENS_PARSED

Definition at line 445 of file MoreLikeThis.cs.

int Lucene.Net.Search.Similar.MoreLikeThis.MaxQueryTerms
getset

Gets or sets the maximum number of query terms that will be included in any generated query. The default is DEFAULT_MAX_QUERY_TERMS.

Definition at line 434 of file MoreLikeThis.cs.

int Lucene.Net.Search.Similar.MoreLikeThis.MaxWordLen
getset

Gets or sets the maximum word length above which words will be ignored. Set this to 0 for no maximum word length. The default is DEFAULT_MAX_WORD_LENGTH.

Definition at line 397 of file MoreLikeThis.cs.

int Lucene.Net.Search.Similar.MoreLikeThis.MinDocFreq
getset

Gets or sets the frequency at which words will be ignored which do not occur in at least this many docs. The default frequency is DEFAULT_MIN_DOC_FREQ.

Definition at line 320 of file MoreLikeThis.cs.

int Lucene.Net.Search.Similar.MoreLikeThis.MinTermFreq
getset

Gets or sets the frequency below which terms will be ignored in the source doc. The default frequency is the DEFAULT_MIN_TERM_FREQ.

Definition at line 310 of file MoreLikeThis.cs.

int Lucene.Net.Search.Similar.MoreLikeThis.MinWordLen
getset

Gets or sets the minimum word length below which words will be ignored. Set this to 0 for no minimum word length. The default is DEFAULT_MIN_WORD_LENGTH.

Definition at line 387 of file MoreLikeThis.cs.

Similarity Lucene.Net.Search.Similar.MoreLikeThis.Similarity
getset

Definition at line 285 of file MoreLikeThis.cs.


The documentation for this class was generated from the following file: