Namespace Lucene.Net.Codecs.Memory

Term dictionary, DocValues or Postings formats that are read entirely into memory.

Classes

DirectDocValuesFormat

In-memory docvalues format that does no (or very little) compression. Indexed values are stored on disk, but then at search time all values are loaded into memory as simple .NET arrays. For numeric values, it uses byte[], short[], int[], long[] as necessary to fit the range of the values. For binary values, there is an int (4 bytes) overhead per value.

Limitations:

  • For binary and sorted fields the total space required for all binary values cannot exceed about 2.1 GB (see MAX_TOTAL_BYTES_LENGTH).
  • For sorted set fields, the sum of the size of each document's set of values cannot exceed about 2.1 B values (see MAX_SORTED_SET_ORDS). For example, if every document has 10 values (10 instances of Lucene.Net.Documents.SortedSetDocValuesField) added, then no more than ~210 M documents can be added to one segment.

DirectPostingsFormat

Wraps Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat format for on-disk storage, but then at read time loads and stores all terms & postings directly in RAM as byte[], int[].

WARNING: This is exceptionally RAM intensive: it makes no effort to compress the postings data, storing terms as separate byte[] and postings as separate int[], but as a result it gives substantial increase in search performance.

This postings format supports Lucene.Net.Index.TermsEnum.Ord and SeekExact(long).

Because this holds all term bytes as a single byte[], you cannot have more than 2.1GB worth of term bytes in a single segment.

Note

This API is experimental and might change in incompatible ways in the next release.

FSTOrdPostingsFormat

FSTOrd term dict + Lucene41PBF

FSTOrdPulsing41PostingsFormat

FSTOrd + Pulsing41

Note

This API is experimental and might change in incompatible ways in the next release.

FSTOrdTermsReader

FST-based terms dictionary reader.

The FST index maps each term and its ord, and during seek the ord is used fetch metadata from a single block. The term dictionary is fully memory resident.

Note

This API is experimental and might change in incompatible ways in the next release.

FSTOrdTermsWriter

FST-based term dict, using ord as FST output.

The FST holds the mapping between <term, ord>, and term's metadata is delta encoded into a single byte block.

Typically the byte block consists of four parts:
  1. term statistics: docFreq, totalTermFreq;
  2. monotonic long[], e.g. the pointer to the postings list for that term;
  3. generic byte[], e.g. other information customized by postings base.
  4. single-level skip list to speed up metadata decoding by ord.

Files:

Term Index

The .tix contains a list of FSTs, one for each field. The FST maps a term to its corresponding order in current field.

  • TermIndex(.tix) --> Header, TermFSTNumFields, Footer
  • TermFST --> Lucene.Net.Util.Fst.FST<T>
  • Header --> CodecHeader (WriteHeader(DataOutput, string, int))
  • Footer --> CodecFooter (Lucene.Net.Codecs.CodecUtil.WriteFooter(Lucene.Net.Store.IndexOutput))

Notes:

  • Since terms are already sorted before writing to Term Block, their ords can directly used to seek term metadata from term block.

Term Block

The .tbk contains all the statistics and metadata for terms, along with field summary (e.g. per-field data like number of documents in current field). For each field, there are four blocks:

  • statistics bytes block: contains term statistics;
  • metadata longs block: delta-encodes monotonic part of metadata;
  • metadata bytes block: encodes other parts of metadata;
  • skip block: contains skip data, to speed up metadata seeking and decoding

File Format:

  • TermBlock(.tbk) --> Header, PostingsHeader, FieldSummary, DirOffset
  • FieldSummary --> NumFields, <FieldNumber, NumTerms, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, DataBlock > NumFields, Footer
  • DataBlock --> StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength, SkipBlock, StatsBlock, MetaLongsBlock, MetaBytesBlock
  • SkipBlock --> < StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipDeltaLongsSize >NumTerms
  • StatsBlock --> < DocFreq[Same?], (TotalTermFreq-DocFreq) ? > NumTerms
  • MetaLongsBlock --> < LongDeltaLongsSize, BytesSize > NumTerms
  • MetaBytesBlock --> Byte MetaBytesBlockLength
  • Header --> CodecHeader (WriteHeader(DataOutput, string, int))
  • DirOffset --> Uint64 (WriteInt64(long))
  • NumFields, FieldNumber, DocCount, DocFreq, LongsSize, FieldNumber, DocCount --> VInt (WriteVInt32(int))
  • NumTerms, SumTotalTermFreq, SumDocFreq, StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength, StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipStart, TotalTermFreq, LongDelta,--> VLong (WriteVInt64(long))
  • Footer --> CodecFooter (Lucene.Net.Codecs.CodecUtil.WriteFooter(Lucene.Net.Store.IndexOutput))

Notes:

  • The format of PostingsHeader and MetaBytes are customized by the specific postings implementation: they contain arbitrary per-file data (such as parameters or versioning information), and per-term data (non-monotonic ones like pulsed postings data).
  • During initialization the reader will load all the blocks into memory. SkipBlock will be decoded, so that during seek term dict can lookup file pointers directly. StatsFPDelta, MetaLongsSkipFPDelta, etc. are file offset for every SkipInterval's term. MetaLongsSkipDelta is the difference from previous one, which indicates the value of preceding metadata longs for every SkipInterval's term.
  • DocFreq is the count of documents which contain the term. TotalTermFreq is the total number of occurrences of the term. Usually these two values are the same for long tail terms, therefore one bit is stole from DocFreq to check this case, so that encoding of TotalTermFreq may be omitted.

Note

This API is experimental and might change in incompatible ways in the next release.

FSTPostingsFormat

FST term dict + Lucene41PBF

FSTPulsing41PostingsFormat

FST + Pulsing41, test only, since FST does no delta encoding here!

Note

This API is experimental and might change in incompatible ways in the next release.

FSTTermsReader

FST-based terms dictionary reader.

The FST directly maps each term and its metadata, it is memory resident.

Note

This API is experimental and might change in incompatible ways in the next release.

FSTTermsWriter

FST-based term dict, using metadata as FST output.

The FST directly holds the mapping between <term, metadata>.

Term metadata consists of three parts:
  1. term statistics: docFreq, totalTermFreq;
  2. monotonic long[], e.g. the pointer to the postings list for that term;
  3. generic byte[], e.g. other information need by postings reader.

File:

Term Dictionary

The .tst contains a list of FSTs, one for each field. The FST maps a term to its corresponding statistics (e.g. docfreq) and metadata (e.g. information for postings list reader like file pointer to postings list).

Typically the metadata is separated into two parts:

  • Monotonical long array: Some metadata will always be ascending in order with the corresponding term. This part is used by FST to share outputs between arcs.
  • Generic byte array: Used to store non-monotonic metadata.

File format:

  • TermsDict(.tst) --> Header, PostingsHeader, FieldSummary, DirOffset
  • FieldSummary --> NumFields, <FieldNumber, NumTerms, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, TermFST >NumFields
  • TermFST TermData
  • TermData --> Flag, BytesSize?, LongDeltaLongsSize?, ByteBytesSize?, < DocFreq[Same?], (TotalTermFreq-DocFreq) > ?
  • Header --> CodecHeader (WriteHeader(DataOutput, string, int))
  • DirOffset --> Uint64 (WriteInt64(long))
  • DocFreq, LongsSize, BytesSize, NumFields, FieldNumber, DocCount --> VInt (WriteVInt32(int))
  • TotalTermFreq, NumTerms, SumTotalTermFreq, SumDocFreq, LongDelta --> VLong (WriteVInt64(long))

Notes:

  • The format of PostingsHeader and generic meta bytes are customized by the specific postings implementation: they contain arbitrary per-file data (such as parameters or versioning information), and per-term data (non-monotonic ones like pulsed postings data).
  • The format of TermData is determined by FST, typically monotonic metadata will be dense around shallow arcs, while in deeper arcs only generic bytes and term statistics exist.
  • The byte Flag is used to indicate which part of metadata exists on current arc. Specially the monotonic part is omitted when it is an array of 0s.
  • Since LongsSize is per-field fixed, it is only written once in field summary.

Note

This API is experimental and might change in incompatible ways in the next release.

MemoryDocValuesFormat

In-memory docvalues format.

MemoryPostingsFormat

Stores terms & postings (docs, positions, payloads) in RAM, using an FST.

Note that this codec implements advance as a linear scan! This means if you store large fields in here, queries that rely on advance will (AND BooleanQuery, PhraseQuery) will be relatively slow!

Note

This API is experimental and might change in incompatible ways in the next release.