The TokenSources type exposes the following members.
Methods
Name | Description | |
---|---|---|
Equals | (Inherited from Object.) | |
Finalize | Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. (Inherited from Object.) | |
GetAnyTokenStream | A convenience method that tries a number of approaches to getting a token stream.
The cost of finding there are no termVectors in the index is minimal (1000 invocations still
registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable
| |
GetHashCode | Serves as a hash function for a particular type. (Inherited from Object.) | |
GetTokenStream(TermPositionVector) | ||
GetTokenStream(TermPositionVector, Boolean) | Low level api.
Returns a token stream or null if no offset info available in index.
This can be used to feed the highlighter with a pre-parsed token stream
In my tests the speeds to recreate 1000 token streams using this method are:
- with TermVector offset only data stored - 420 milliseconds
- with TermVector offset AND position data stored - 271 milliseconds
(nb timings for TermVector with position data are based on a tokenizer with contiguous
positions - no overlaps or gaps)
The cost of not using TermPositionVector to store
pre-parsed content and using an analyzer to re-parse the original content:
- reanalyzing the original content - 980 milliseconds
The re-analyze timings will typically vary depending on -
1) The complexity of the analyzer code (timings above were using a
stemmer/lowercaser/stopword combo)
2) The number of other fields (Lucene reads ALL fields off the disk
when accessing just one document field - can cost dear!)
3) Use of compression on field storage - could be faster cos of compression (less disk IO)
or slower (more CPU burn) depending on the content.
| |
GetTokenStream(IndexReader, Int32, String) | ||
GetTokenStream(IndexReader, Int32, String, Analyzer) | ||
GetType | Gets the Type of the current instance. (Inherited from Object.) | |
MemberwiseClone | Creates a shallow copy of the current Object. (Inherited from Object.) | |
ToString | (Inherited from Object.) |