Class FSTOrdTermsWriter

FST-based term dict, using ord as FST output.

The FST holds the mapping between <term, ord>, and term's metadata is delta encoded into a single byte block.

Typically the byte block consists of four parts:

  1. term statistics: docFreq, totalTermFreq;
  2. monotonic long[], e.g. the pointer to the postings list for that term;
  3. generic byte[], e.g. other information customized by postings base.
  4. single-level skip list to speed up metadata decoding by ord.

Files:

Term Index

The .tix contains a list of FSTs, one for each field. The FST maps a term to its corresponding order in current field.

Notes:

  • Since terms are already sorted before writing to Term Block, their ords can directly used to seek term metadata from term block.

Term Block

The .tbk contains all the statistics and metadata for terms, along with field summary (e.g. per-field data like number of documents in current field). For each field, there are four blocks:

  • statistics bytes block: contains term statistics;
  • metadata longs block: delta-encodes monotonic part of metadata;
  • metadata bytes block: encodes other parts of metadata;
  • skip block: contains skip data, to speed up metadata seeking and decoding

File Format:

  • TermBlock(.tbk) --> Header, PostingsHeader, FieldSummary, DirOffset
  • FieldSummary --> NumFields, <FieldNumber, NumTerms, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, DataBlock > NumFields, Footer
  • DataBlock --> StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength, SkipBlock, StatsBlock, MetaLongsBlock, MetaBytesBlock
  • SkipBlock --> < StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipDeltaLongsSize >NumTerms
  • StatsBlock --> < DocFreq[Same?], (TotalTermFreq-DocFreq) ? > NumTerms
  • MetaLongsBlock --> < LongDeltaLongsSize, BytesSize > NumTerms
  • MetaBytesBlock --> Byte MetaBytesBlockLength
  • Header --> CodecHeader (WriteHeader(DataOutput, String, Int32))
  • DirOffset --> Uint64 (WriteInt64(Int64))
  • NumFields, FieldNumber, DocCount, DocFreq, LongsSize, FieldNumber, DocCount --> VInt (WriteVInt32(Int32))
  • NumTerms, SumTotalTermFreq, SumDocFreq, StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength, StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipStart, TotalTermFreq, LongDelta,--> VLong (WriteVInt64(Int64))
  • Footer --> CodecFooter (WriteFooter(IndexOutput))

Notes:

  • The format of PostingsHeader and MetaBytes are customized by the specific postings implementation: they contain arbitrary per-file data (such as parameters or versioning information), and per-term data (non-monotonic ones like pulsed postings data).
  • During initialization the reader will load all the blocks into memory. SkipBlock will be decoded, so that during seek term dict can lookup file pointers directly. StatsFPDelta, MetaLongsSkipFPDelta, etc. are file offset for every SkipInterval's term. MetaLongsSkipDelta is the difference from previous one, which indicates the value of preceding metadata longs for every SkipInterval's term.
  • DocFreq is the count of documents which contain the term. TotalTermFreq is the total number of occurrences of the term. Usually these two values are the same for long tail terms, therefore one bit is stole from DocFreq to check this case, so that encoding of TotalTermFreq may be omitted.

This is a Lucene.NET EXPERIMENTAL API, use at your own risk

Inheritance
System.Object
Lucene.Net.Codecs.FieldsConsumer
FSTOrdTermsWriter
Implements
System.IDisposable
Inherited Members
Lucene.Net.Codecs.FieldsConsumer.Dispose()
Lucene.Net.Codecs.FieldsConsumer.Merge(Lucene.Net.Index.MergeState, Lucene.Net.Index.Fields)
System.Object.Equals(System.Object)
System.Object.Equals(System.Object, System.Object)
System.Object.GetHashCode()
System.Object.GetType()
System.Object.MemberwiseClone()
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.ToString()
Namespace: Lucene.Net.Codecs.Memory
Assembly: Lucene.Net.Codecs.dll
Syntax
public class FSTOrdTermsWriter : FieldsConsumer, IDisposable

Constructors

| Improve this Doc View Source

FSTOrdTermsWriter(SegmentWriteState, PostingsWriterBase)

Declaration
public FSTOrdTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter)
Parameters
Type Name Description
Lucene.Net.Index.SegmentWriteState state
Lucene.Net.Codecs.PostingsWriterBase postingsWriter

Fields

| Improve this Doc View Source

SKIP_INTERVAL

Declaration
public const int SKIP_INTERVAL = 8
Field Value
Type Description
System.Int32
| Improve this Doc View Source

TERMS_VERSION_CHECKSUM

Declaration
public const int TERMS_VERSION_CHECKSUM = 1
Field Value
Type Description
System.Int32
| Improve this Doc View Source

TERMS_VERSION_CURRENT

Declaration
public const int TERMS_VERSION_CURRENT = 1
Field Value
Type Description
System.Int32
| Improve this Doc View Source

TERMS_VERSION_START

Declaration
public const int TERMS_VERSION_START = 0
Field Value
Type Description
System.Int32

Methods

| Improve this Doc View Source

AddField(FieldInfo)

Declaration
public override TermsConsumer AddField(FieldInfo field)
Parameters
Type Name Description
Lucene.Net.Index.FieldInfo field
Returns
Type Description
Lucene.Net.Codecs.TermsConsumer
Overrides
Lucene.Net.Codecs.FieldsConsumer.AddField(Lucene.Net.Index.FieldInfo)
| Improve this Doc View Source

Dispose(Boolean)

Declaration
protected override void Dispose(bool disposing)
Parameters
Type Name Description
System.Boolean disposing
Overrides

Implements

System.IDisposable