Class TokenSources

Hides implementation issues associated with obtaining a Lucene.Net.Analysis.TokenStream for use with the Highlighter - can obtain from term vectors with offsets and positions or from an Analyzer re-parsing the stored content. see TokenStreamFromTermVector

Inheritance

System.Object

TokenSources

Inherited Members

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Namespace: Lucene.Net.Search.Highlight

Assembly: Lucene.Net.Highlighter.dll

Syntax

public static class TokenSources

Methods

| Improve this Doc View Source

GetAnyTokenStream(IndexReader, Int32, String, Analyzer)

A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable

Declaration

public static TokenStream GetAnyTokenStream(IndexReader reader, int docId, string field, Analyzer analyzer)

Parameters

Type	Name	Description
Lucene.Net.Index.IndexReader	reader
System.Int32	docId
System.String	field
Lucene.Net.Analysis.Analyzer	analyzer

Returns

Type	Description
Lucene.Net.Analysis.TokenStream	null if field not stored correctly

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error

| Improve this Doc View Source

GetAnyTokenStream(IndexReader, Int32, String, Document, Analyzer)

A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to using the passed in Lucene.Net.Documents.Document to retrieve the Lucene.Net.Analysis.TokenStream. This is useful when you already have the document, but would prefer to use the vector first.

Declaration

public static TokenStream GetAnyTokenStream(IndexReader reader, int docId, string field, Document doc, Analyzer analyzer)

Parameters

Type	Name	Description
Lucene.Net.Index.IndexReader	reader	The Lucene.Net.Index.IndexReader to use to try and get the vector from
System.Int32	docId	The docId to retrieve.
System.String	field	The field to retrieve on the document
Lucene.Net.Documents.Document	doc	The document to fall back on
Lucene.Net.Analysis.Analyzer	analyzer	The analyzer to use for creating the TokenStream if the vector doesn't exist

Returns

Type	Description
Lucene.Net.Analysis.TokenStream	The Lucene.Net.Analysis.TokenStream for the Lucene.Net.Index.IIndexableField on the Lucene.Net.Documents.Document

Exceptions

Type	Condition
System.IO.IOException	if there was an error loading

| Improve this Doc View Source

GetTokenStream(Document, String, Analyzer)

Declaration

public static TokenStream GetTokenStream(Document doc, string field, Analyzer analyzer)

Parameters

Type	Name	Description
Lucene.Net.Documents.Document	doc
System.String	field
Lucene.Net.Analysis.Analyzer	analyzer

Returns

Type	Description
Lucene.Net.Analysis.TokenStream

| Improve this Doc View Source

GetTokenStream(IndexReader, Int32, String, Analyzer)

Declaration

public static TokenStream GetTokenStream(IndexReader reader, int docId, string field, Analyzer analyzer)

Parameters

Type	Name	Description
Lucene.Net.Index.IndexReader	reader
System.Int32	docId
System.String	field
Lucene.Net.Analysis.Analyzer	analyzer

Returns

Type	Description
Lucene.Net.Analysis.TokenStream

| Improve this Doc View Source

GetTokenStream(Terms)

Declaration

public static TokenStream GetTokenStream(Terms vector)

Parameters

Type	Name	Description
Lucene.Net.Index.Terms	vector

Returns

Type	Description
Lucene.Net.Analysis.TokenStream

| Improve this Doc View Source

GetTokenStream(Terms, Boolean)

Low level api. Returns a token stream generated from a Lucene.Net.Index.Terms. This can be used to feed the highlighter with a pre-parsed token stream. The Lucene.Net.Index.Terms must have offsets available.

In my tests the speeds to recreate 1000 token streams using this method are:

with TermVector offset only data stored - 420 milliseconds
with TermVector offset AND position data stored - 271 milliseconds (nb timings for TermVector with position data are based on a tokenizer with contiguous positions - no overlaps or gaps)
The cost of not using TermPositionVector to store pre-parsed content and using an analyzer to re-parse the original content:
- reanalyzing the original content - 980 milliseconds

The re-analyze timings will typically vary depending on -

The complexity of the analyzer code (timings above were using a stemmer/lowercaser/stopword combo)
The number of other fields (Lucene reads ALL fields off the disk when accessing just one document field - can cost dear!)
Use of compression on field storage - could be faster due to compression (less disk IO) or slower (more CPU burn) depending on the content.

Declaration

public static TokenStream GetTokenStream(Terms tpv, bool tokenPositionsGuaranteedContiguous)

Parameters

Type	Name	Description
Lucene.Net.Index.Terms	tpv
System.Boolean	tokenPositionsGuaranteedContiguous	true if the token position numbers have no overlaps or gaps. If looking to eek out the last drops of performance, set to true. If in doubt, set to false.

Returns

Type	Description
Lucene.Net.Analysis.TokenStream

Exceptions

Type	Condition
System.ArgumentException	if no offsets are available

| Improve this Doc View Source

GetTokenStream(String, String, Analyzer)

Declaration

public static TokenStream GetTokenStream(string field, string contents, Analyzer analyzer)

Parameters

Type	Name	Description
System.String	field
System.String	contents
Lucene.Net.Analysis.Analyzer	analyzer

Returns

Type	Description
Lucene.Net.Analysis.TokenStream

| Improve this Doc View Source

GetTokenStreamWithOffsets(IndexReader, Int32, String)

Returns a Lucene.Net.Analysis.TokenStream with positions and offsets constructed from field termvectors. If the field has no termvectors or offsets are not included in the termvector, return null. See GetTokenStream(Terms) for an explanation of what happens when positions aren't present.

Declaration

public static TokenStream GetTokenStreamWithOffsets(IndexReader reader, int docId, string field)

Parameters

Type	Name	Description
Lucene.Net.Index.IndexReader	reader	the Lucene.Net.Index.IndexReader to retrieve term vectors from
System.Int32	docId	the document to retrieve term vectors for
System.String	field	the field to retrieve term vectors for

Returns

Type	Description
Lucene.Net.Analysis.TokenStream	a Lucene.Net.Analysis.TokenStream, or null if offsets are not available

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error