Namespace Lucene.Net.Codecs

Codecs API: API for customization of the encoding and structure of the index.

The Codec API allows you to customise the way the following pieces of index information are stored: * Postings lists - see PostingsFormat * DocValues - see DocValuesFormat * Stored fields - see StoredFieldsFormat * Term vectors - see TermVectorsFormat * FieldInfos - see FieldInfosFormat * SegmentInfo - see SegmentInfoFormat * Norms - see NormsFormat * Live documents - see LiveDocsFormat

For some concrete implementations beyond Lucene's official index format, see the Codecs module.

Codecs are identified by name through the Java Service Provider Interface. To create your own codec, extend Codec and pass the new codec's name to the super() constructor: public class MyCodec extends Codec { public MyCodec() { super("MyCodecName"); } ... } You will need to register the Codec class so that the {@link java.util.ServiceLoader ServiceLoader} can find it, by including a META-INF/services/org.apache.lucene.codecs.Codec file on your classpath that contains the package-qualified name of your codec.

If you just want to customise the PostingsFormat, or use different postings formats for different fields, then you can register your custom postings format in the same way (in META-INF/services/org.apache.lucene.codecs.PostingsFormat), and then extend the default Lucene46Codec and override #getPostingsFormatForField(String) to return your custom postings format.

Similarly, if you just want to customise the DocValuesFormat per-field, have a look at #getDocValuesFormatForField(String).

Classes

BlockTermState

Holds all state required for PostingsReaderBase to produce a DocsEnum without re-seeking the terms dict.

BlockTreeTermsReader

A block-based terms index and dictionary that assigns terms to variable length blocks according to how they share prefixes. The terms index is a prefix trie whose leaves are term blocks. The advantage of this approach is that SeekExact() is often able to determine a term cannot exist without doing any IO, and intersection with Automata is very fast. Note that this terms dictionary has it's own fixed terms index (ie, it does not support a pluggable terms index implementation).

NOTE: this terms dictionary does not support index divisor when opening an IndexReader. Instead, you can change the min/maxItemsPerBlock during indexing.

The data structure used by this implementation is very similar to a burst trie (http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.3499), but with added logic to break up too-large blocks of all terms sharing a given prefix into smaller ones.

Use CheckIndex with the -verbose option to see summary statistics on the blocks in the dictionary.

See BlockTreeTermsWriter.

Note

This API is experimental and might change in incompatible ways in the next release.

BlockTreeTermsReader.FieldReader

BlockTree's implementation of GetTerms(String).

BlockTreeTermsReader.Stats

BlockTree statistics for a single field returned by ComputeStats().

BlockTreeTermsWriter

Block-based terms index and dictionary writer.

Writes terms dict and index, block-encoding (column stride) each term's metadata for each set of terms between two index terms.

Files:

Term Dictionary

The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and per-term metadata (typically pointers to the postings list for that term in the inverted index).

The .tim is arranged in blocks: with blocks containing a variable number of entries (by default 25-48), where each entry is either a term or a reference to a sub-block.

NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the Postings Metadata and Term Metadata sections.

  • TermsDict (.tim) --> Header, PostingsHeader, NodeBlockNumBlocks, FieldSummary, DirOffset, Footer
  • NodeBlock --> (OuterNode | InnerNode)
  • OuterNode --> EntryCount, SuffixLength, ByteSuffixLength, StatsLength, < TermStats >EntryCount, MetaLength, <TermMetadata>EntryCount
  • InnerNode --> EntryCount, SuffixLength[,Sub?], ByteSuffixLength, StatsLength, < TermStats ? >EntryCount, MetaLength, <TermMetadata ? >EntryCount
  • TermStats --> DocFreq, TotalTermFreq
  • FieldSummary --> NumFields, <FieldNumber, NumTerms, RootCodeLength, ByteRootCodeLength, SumTotalTermFreq?, SumDocFreq, DocCount>NumFields
  • Header --> CodecHeader (WriteHeader(DataOutput, String, Int32)
  • DirOffset --> Uint64 (WriteInt64(Int64))
  • EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength,NumFields, FieldNumber,RootCodeLength,DocCount --> VInt (WriteVInt32(Int32)_
  • TotalTermFreq,NumTerms,SumTotalTermFreq,SumDocFreq --> VLong (WriteVInt64(Int64))
  • Footer --> CodecFooter (WriteFooter(IndexOutput))

Notes:

  • Header is a CodecHeader (WriteHeader(DataOutput, String, Int32)) storing the version information for the BlockTree implementation.
  • DirOffset is a pointer to the FieldSummary section.
  • DocFreq is the count of documents which contain the term.
  • TotalTermFreq is the total number of occurrences of the term. this is encoded as the difference between the total number of occurrences and the DocFreq.
  • FieldNumber is the fields number from Lucene.Net.Codecs.BlockTreeTermsWriter.fieldInfos. (.fnm)
  • NumTerms is the number of unique terms for the field.
  • RootCode points to the root block for the field.
  • SumDocFreq is the total number of postings, the number of term-document pairs across the entire field.
  • DocCount is the number of documents that have at least one posting for this field.
  • PostingsHeader and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files).
  • For inner nodes of the tree, every entry will steal one bit to mark whether it points to child nodes(sub-block). If so, the corresponding TermStats and TermMetadata are omitted

Term Index

The .tip file contains an index into the term dictionary, so that it can be accessed randomly. The index is also used to determine when a given term cannot exist on disk (in the .tim file), saving a disk seek.

Notes:

  • The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST.
  • DirOffset is a pointer to the start of the IndexStartFPs for all fields
  • It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.

Note

This API is experimental and might change in incompatible ways in the next release.

Codec

Encodes/decodes an inverted index segment.

Note, when extending this class, the name (Name) is written into the index. In order for the segment to be read, the name must resolve to your implementation via ForName(String). This method uses GetCodec(String) to resolve codec names.

To implement your own codec:

  1. Subclass this class.
  2. Subclass DefaultCodecFactory, override the Initialize() method, and add the line base.ScanForCodecs(typeof(YourCodec).Assembly). If you have any codec classes in your assembly that are not meant for reading, you can add the ExcludeCodecFromScanAttribute to them so they are ignored by the scan.
  3. set the new ICodecFactory by calling SetCodecFactory(ICodecFactory) at application startup.
If your codec has dependencies, you may also override GetCodec(Type) to inject them via pure DI or a DI container. See DI-Friendly Framework to understand the approach used.

Codec Names

Unlike the Java version, codec names are by default convention-based on the class name. If you name your custom codec class "MyCustomCodec", the codec name will the same name without the "Codec" suffix: "MyCustom".

You can override this default behavior by using the CodecNameAttribute to name the codec differently than this convention. Codec names must be all ASCII alphanumeric, and less than 128 characters in length.

CodecNameAttribute

Represents an attribute that is used to name a Codec, if a name other than the default Codec naming convention is desired.

CodecUtil

Utility class for reading and writing versioned headers.

Writing codec headers is useful to ensure that a file is in the format you think it is.

Note

This API is experimental and might change in incompatible ways in the next release.

DefaultCodecFactory

Implements the default functionality of ICodecFactory.

To replace the DefaultCodecFactory instance, call SetCodecFactory(ICodecFactory) at application start up. DefaultCodecFactory can be subclassed or passed additional parameters to register additional codecs, inject dependencies, or change caching behavior, as shown in the following examples. Alternatively, ICodecFactory can be implemented to provide complete control over codec creation and lifetimes.

Register Additional Codecs

Additional codecs can be added by initializing the instance of DefaultCodecFactory and passing an array of Codec-derived types.

// Register the factory at application start up.
Codec.SetCodecFactory(new DefaultCodecFactory {
    CustomCodecTypes = new Type[] { typeof(MyCodec), typeof(AnotherCodec) }
});

Only Use Explicitly Defined Codecs

PutCodecType(Type) can be used to explicitly add codec types. In this example, the call to base.Initialize() is excluded to skip the built-in codec registration. Since AnotherCodec doesn't have a default constructor, the NewCodec(Type) method is overridden to supply the required parameters.

public class ExplicitCodecFactory : DefaultCodecFactory
{
    protected override void Initialize()
    {
        // Load specific codecs in a specific order.
        PutCodecType(typeof(MyCodec));
        PutCodecType(typeof(AnotherCodec));
    }

    protected override Codec NewCodec(Type type)
    {
        // Special case: AnotherCodec has a required dependency
        if (typeof(AnotherCodec).Equals(type))
            return new AnotherCodec(new SomeDependency());

        return base.NewCodec(type);
    }
}

// Register the factory at application start up.
Codec.SetCodecFactory(new ExplicitCodecFactory());

See the Lucene.Net.Codecs namespace documentation for more examples of how to inject dependencies into Codec subclasses.

Use Reflection to Scan an Assembly for Codecs

ScanForCodecs(Assembly) or ScanForCodecs(IEnumerable<Assembly>) can be used to scan assemblies using .NET Reflection for codec types and add all subclasses that are found automatically. This example calls base.Initialize() to load the default codecs prior to scanning for additional codecs.

public class ScanningCodecFactory : DefaultCodecFactory
{
    protected override void Initialize()
    {
        // Load all default codecs
        base.Initialize();

        // Load all of the codecs inside of the same assembly that MyCodec is defined in
        ScanForCodecs(typeof(MyCodec).Assembly);
    }
}

// Register the factory at application start up.
Codec.SetCodecFactory(new ScanningCodecFactory());

Codecs in the target assemblie(s) can be excluded from the scan by decorating them with the ExcludeCodecFromScanAttribute.

DefaultDocValuesFormatFactory

Implements the default functionality of IDocValuesFormatFactory.

To replace the DefaultDocValuesFormatFactory instance, call SetDocValuesFormatFactory(IDocValuesFormatFactory) at application start up. DefaultDocValuesFormatFactory can be subclassed or passed additional parameters to register additional codecs, inject dependencies, or change caching behavior, as shown in the following examples. Alternatively, IDocValuesFormatFactory can be implemented to provide complete control over doc values format creation and lifetimes.

Register Additional DocValuesFormats

Additional codecs can be added by initializing the instance of DefaultDocValuesFormatFactory and passing an array of DocValuesFormat-derived types.

// Register the factory at application start up.
DocValuesFormat.SetDocValuesFormatFactory(new DefaultDocValuesFormatFactory {
    CustomDocValuesFormatTypes = new Type[] { typeof(MyDocValuesFormat), typeof(AnotherDocValuesFormat) }
});

Only Use Explicitly Defined DocValuesFormats

PutDocValuesFormatType(Type) can be used to explicitly add codec types. In this example, the call to base.Initialize() is excluded to skip the built-in codec registration. Since AnotherDocValuesFormat doesn't have a default constructor, the NewDocValuesFormat(Type) method is overridden to supply the required parameters.

public class ExplicitDocValuesFormatFactory : DefaultDocValuesFormatFactory
{
    protected override void Initialize()
    {
        // Load specific codecs in a specific order.
        PutDocValuesFormatType(typeof(MyDocValuesFormat));
        PutDocValuesFormatType(typeof(AnotherDocValuesFormat));
    }

    protected override DocValuesFormat NewDocValuesFormat(Type type)
    {
        // Special case: AnotherDocValuesFormat has a required dependency
        if (typeof(AnotherDocValuesFormat).Equals(type))
            return new AnotherDocValuesFormat(new SomeDependency());

        return base.NewDocValuesFormat(type);
    }
}

// Register the factory at application start up.
DocValuesFormat.SetDocValuesFormatFactory(new ExplicitDocValuesFormatFactory());

See the Lucene.Net.Codecs namespace documentation for more examples of how to inject dependencies into DocValuesFormat subclasses.

Use Reflection to Scan an Assembly for DocValuesFormats

ScanForDocValuesFormats(Assembly) or ScanForDocValuesFormats(IEnumerable<Assembly>) can be used to scan assemblies using .NET Reflection for codec types and add all subclasses that are found automatically.

public class ScanningDocValuesFormatFactory : DefaultDocValuesFormatFactory
{
    protected override void Initialize()
    {
        // Load all default codecs
        base.Initialize();

        // Load all of the codecs inside of the same assembly that MyDocValuesFormat is defined in
        ScanForDocValuesFormats(typeof(MyDocValuesFormat).Assembly);
    }
}

// Register the factory at application start up.
DocValuesFormat.SetDocValuesFormatFactory(new ScanningDocValuesFormatFactory());

Doc values formats in the target assembly can be excluded from the scan by decorating them with the ExcludeDocValuesFormatFromScanAttribute.

DefaultPostingsFormatFactory

Implements the default functionality of IPostingsFormatFactory.

To replace the DefaultPostingsFormatFactory instance, call SetPostingsFormatFactory(IPostingsFormatFactory) at application start up. DefaultPostingsFormatFactory can be subclassed or passed additional parameters to register additional codecs, inject dependencies, or change caching behavior, as shown in the following examples. Alternatively, IPostingsFormatFactory can be implemented to provide complete control over postings format creation and lifetimes.

Register Additional PostingsFormats

Additional codecs can be added by initializing the instance of DefaultPostingsFormatFactory and passing an array of PostingsFormat-derived types.

// Register the factory at application start up.
PostingsFormat.SetPostingsFormatFactory(new DefaultPostingsFormatFactory {
    CustomPostingsFormatTypes = new Type[] { typeof(MyPostingsFormat), typeof(AnotherPostingsFormat) }
});

Only Use Explicitly Defined PostingsFormats

PutPostingsFormatType(Type) can be used to explicitly add codec types. In this example, the call to base.Initialize() is excluded to skip the built-in codec registration. Since AnotherPostingsFormat doesn't have a default constructor, the NewPostingsFormat(Type) method is overridden to supply the required parameters.

public class ExplicitPostingsFormatFactory : DefaultPostingsFormatFactory
{
    protected override void Initialize()
    {
        // Load specific codecs in a specific order.
        PutPostingsFormatType(typeof(MyPostingsFormat));
        PutPostingsFormatType(typeof(AnotherPostingsFormat));
    }

    protected override PostingsFormat NewPostingsFormat(Type type)
    {
        // Special case: AnotherPostingsFormat has a required dependency
        if (typeof(AnotherPostingsFormat).Equals(type))
            return new AnotherPostingsFormat(new SomeDependency());

        return base.NewPostingsFormat(type);
    }
}

// Register the factory at application start up.
PostingsFormat.SetPostingsFormatFactory(new ExplicitPostingsFormatFactory());

See the Lucene.Net.Codecs namespace documentation for more examples of how to inject dependencies into PostingsFormat subclasses.

Use Reflection to Scan an Assembly for PostingsFormats

ScanForPostingsFormats(Assembly) or ScanForPostingsFormats(IEnumerable<Assembly>) can be used to scan assemblies using .NET Reflection for codec types and add all subclasses that are found automatically.

public class ScanningPostingsFormatFactory : DefaultPostingsFormatFactory
{
    protected override void Initialize()
    {
        // Load all default codecs
        base.Initialize();

        // Load all of the codecs inside of the same assembly that MyPostingsFormat is defined in
        ScanForPostingsFormats(typeof(MyPostingsFormat).Assembly);
    }
}

// Register the factory at application start up.
PostingsFormat.SetPostingsFormatFactory(new ScanningPostingsFormatFactory());

Postings formats in the target assembly can be excluded from the scan by decorating them with the ExcludePostingsFormatFromScanAttribute.

DocValuesConsumer

Abstract API that consumes numeric, binary and sorted docvalues. Concrete implementations of this actually do "something" with the docvalues (write it into the index in a specific format).

The lifecycle is:

  1. DocValuesConsumer is created by FieldsConsumer(SegmentWriteState) or NormsConsumer(SegmentWriteState).
  2. AddNumericField(FieldInfo, IEnumerable<Nullable<Int64>>), AddBinaryField(FieldInfo, IEnumerable<BytesRef>), or AddSortedField(FieldInfo, IEnumerable<BytesRef>, IEnumerable<Nullable<Int64>>) are called for each Numeric, Binary, or Sorted docvalues field. The API is a "pull" rather than "push", and the implementation is free to iterate over the values multiple times (System.Collections.Generic.IEnumerable<T>.GetEnumerator()).
  3. After all fields are added, the consumer is Dispose()d.

Note

This API is experimental and might change in incompatible ways in the next release.

DocValuesFormat

Encodes/decodes per-document values.

Note, when extending this class, the name (Name) may written into the index in certain configurations. In order for the segment to be read, the name must resolve to your implementation via ForName(String). This method uses GetDocValuesFormat(String) to resolve format names.

To implement your own format:

  1. Subclass this class.
  2. Subclass DefaultDocValuesFormatFactory, override the Initialize() method, and add the line base.ScanForDocValuesFormats(typeof(YourDocValuesFormat).Assembly). If you have any format classes in your assembly that are not meant for reading, you can add the ExcludeDocValuesFormatFromScanAttribute to them so they are ignored by the scan.
  3. Set the new IDocValuesFormatFactory by calling SetDocValuesFormatFactory(IDocValuesFormatFactory) at application startup.
If your format has dependencies, you may also override GetDocValuesFormat(Type) to inject them via pure DI or a DI container. See DI-Friendly Framework to understand the approach used.

DocValuesFormat Names

Unlike the Java version, format names are by default convention-based on the class name. If you name your custom format class "MyCustomDocValuesFormat", the format name will the same name without the "DocValuesFormat" suffix: "MyCustom".

You can override this default behavior by using the DocValuesFormatNameAttribute to name the format differently than this convention. Format names must be all ASCII alphanumeric, and less than 128 characters in length.

Note

This API is experimental and might change in incompatible ways in the next release.

DocValuesFormatNameAttribute

Represents an attribute that is used to name a DocValuesFormat, if a name other than the default DocValuesFormat naming convention is desired.

DocValuesProducer

Abstract API that produces numeric, binary and sorted docvalues.

Note

This API is experimental and might change in incompatible ways in the next release.

ExcludeCodecFromScanAttribute

When placed on a class that subclasses Codec, adding this attribute will exclude the type from consideration in the ScanForCodecs(Assembly) method.

However, the Codec type can still be added manually using PutCodecType(Type).

ExcludeDocValuesFormatFromScanAttribute

When placed on a class that subclasses DocValuesFormat, adding this attribute will exclude the type from consideration in the ScanForDocValuesFormats(Assembly) method.

However, the DocValuesFormat type can still be added manually using PutDocValuesFormatType(Type).

ExcludePostingsFormatFromScanAttribute

When placed on a class that subclasses PostingsFormat, adding this attribute will exclude the type from consideration in the ScanForPostingsFormats(Assembly) method.

However, the PostingsFormat type can still be added manually using PutPostingsFormatType(Type).

FieldInfosFormat

Encodes/decodes FieldInfos.

Note

This API is experimental and might change in incompatible ways in the next release.

FieldInfosReader

Codec API for reading FieldInfos.

Note

This API is experimental and might change in incompatible ways in the next release.

FieldInfosWriter

Codec API for writing FieldInfos.

Note

This API is experimental and might change in incompatible ways in the next release.

FieldsConsumer

Abstract API that consumes terms, doc, freq, prox, offset and payloads postings. Concrete implementations of this actually do "something" with the postings (write it into the index in a specific format).

The lifecycle is:

  1. FieldsConsumer is created by FieldsConsumer(SegmentWriteState).
  2. For each field, AddField(FieldInfo) is called, returning a TermsConsumer for the field.
  3. After all fields are added, the consumer is Dispose()d.

Note

This API is experimental and might change in incompatible ways in the next release.

FieldsProducer

Abstract API that produces terms, doc, freq, prox, offset and payloads postings.

Note

This API is experimental and might change in incompatible ways in the next release.

FilterCodec

A codec that forwards all its method calls to another codec.

Extend this class when you need to reuse the functionality of an existing codec. For example, if you want to build a codec that redefines Lucene46's LiveDocsFormat:

    public sealed class CustomCodec : FilterCodec 
    {
        public CustomCodec()
            : base("CustomCodec", new Lucene46Codec())
        {
        }

        public override LiveDocsFormat LiveDocsFormat 
        {
            get { return new CustomLiveDocsFormat(); }
        }
    }

Please note: Don't call ForName(String) from the no-arg constructor of your own codec. When the DefaultCodecFactory loads your own Codec, the DefaultCodecFactory has not yet fully initialized! If you want to extend another Codec, instantiate it directly by calling its constructor.

Note

This API is experimental and might change in incompatible ways in the next release.

LiveDocsFormat

Format for live/deleted documents.

Note

This API is experimental and might change in incompatible ways in the next release.

MappingMultiDocsAndPositionsEnum

Exposes flex API, merged from flex API of sub-segments, remapping docIDs (this is used for segment merging).

Note

This API is experimental and might change in incompatible ways in the next release.

MappingMultiDocsEnum

Exposes flex API, merged from flex API of sub-segments, remapping docIDs (this is used for segment merging).

Note

This API is experimental and might change in incompatible ways in the next release.

MultiLevelSkipListReader

This abstract class reads skip lists with multiple levels.

See MultiLevelSkipListWriter for the information about the encoding of the multi level skip lists.

Subclasses must implement the abstract method ReadSkipData(Int32, IndexInput) which defines the actual format of the skip data.

Note

This API is experimental and might change in incompatible ways in the next release.

MultiLevelSkipListWriter

This abstract class writes skip lists with multiple levels.

Example for skipInterval = 3:
                                                    c            (skip level 2)
                c                 c                 c            (skip level 1)
    x     x     x     x     x     x     x     x     x     x      (skip level 0)
d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d  (posting list)
    3     6     9     12    15    18    21    24    27    30     (df)

d - document
x - skip data
c - skip data with child pointer

Skip level i contains every skipInterval-th entry from skip level i-1.
Therefore the number of entries on level i is: floor(df / ((skipInterval ^ (i + 1))).

Each skip entry on a level i>0 contains a pointer to the corresponding skip entry in list i-1.
this guarantees a logarithmic amount of skips to find the target document.

While this class takes care of writing the different skip levels,
subclasses must define the actual format of the skip data.

Note

This API is experimental and might change in incompatible ways in the next release.

NormsFormat

Encodes/decodes per-document score normalization values.

PostingsBaseFormat

Provides a PostingsReaderBase and PostingsWriterBase.

Note

This API is experimental and might change in incompatible ways in the next release.

PostingsConsumer

Abstract API that consumes postings for an individual term.

The lifecycle is:

  1. PostingsConsumer is returned for each term by StartTerm(BytesRef).
  2. StartDoc(Int32, Int32) is called for each document where the term occurs, specifying id and term frequency for that document.
  3. If positions are enabled for the field, then AddPosition(Int32, BytesRef, Int32, Int32) will be called for each occurrence in the document.
  4. FinishDoc() is called when the producer is done adding positions to the document.

Note

This API is experimental and might change in incompatible ways in the next release.

PostingsFormat

Encodes/decodes terms, postings, and proximity data.

Note, when extending this class, the name (Name) may written into the index in certain configurations. In order for the segment to be read, the name must resolve to your implementation via ForName(String). This method uses GetPostingsFormat(String) to resolve format names.

If you implement your own format:

  1. Subclass this class.
  2. Subclass DefaultPostingsFormatFactory, override Initialize(), and add the line base.ScanForPostingsFormats(typeof(YourPostingsFormat).Assembly). If you have any format classes in your assembly that are not meant for reading, you can add the ExcludePostingsFormatFromScanAttribute to them so they are ignored by the scan.
  3. Set the new IPostingsFormatFactory by calling SetPostingsFormatFactory(IPostingsFormatFactory) at application startup.
If your format has dependencies, you may also override GetPostingsFormat(Type) to inject them via pure DI or a DI container. See DI-Friendly Framework to understand the approach used.

PostingsFormat Names

Unlike the Java version, format names are by default convention-based on the class name. If you name your custom format class "MyCustomPostingsFormat", the codec name will the same name without the "PostingsFormat" suffix: "MyCustom".

You can override this default behavior by using the PostingsFormatNameAttribute to name the format differently than this convention. Format names must be all ASCII alphanumeric, and less than 128 characters in length.

Note

This API is experimental and might change in incompatible ways in the next release.

PostingsFormatNameAttribute

Represents an attribute that is used to name a PostingsFormat, if a name other than the default PostingsFormat naming convention is desired.

PostingsReaderBase

The core terms dictionaries (BlockTermsReader, BlockTreeTermsReader) interact with a single instance of this class to manage creation of DocsEnum and DocsAndPositionsEnum instances. It provides an IndexInput (termsIn) where this class may read any previously stored data that it had written in its corresponding PostingsWriterBase at indexing time.

Note

This API is experimental and might change in incompatible ways in the next release.

PostingsWriterBase

Extension of PostingsConsumer to support pluggable term dictionaries.

This class contains additional hooks to interact with the provided term dictionaries such as BlockTreeTermsWriter. If you want to re-use an existing implementation and are only interested in customizing the format of the postings list, extend this class instead.

Note

This API is experimental and might change in incompatible ways in the next release.

SegmentInfoFormat

Expert: Controls the format of the SegmentInfo (segment metadata file).

Note

This API is experimental and might change in incompatible ways in the next release.

SegmentInfoReader

Specifies an API for classes that can read SegmentInfo information.

Note

This API is experimental and might change in incompatible ways in the next release.

SegmentInfoWriter

Specifies an API for classes that can write out SegmentInfo data.

Note

This API is experimental and might change in incompatible ways in the next release.

StoredFieldsFormat

Controls the format of stored fields.

StoredFieldsReader

Codec API for reading stored fields.

You need to implement VisitDocument(Int32, StoredFieldVisitor) to read the stored fields for a document, implement Clone() (creating clones of any IndexInputs used, etc), and Dispose(Boolean) to cleanup any allocated resources.

Note

This API is experimental and might change in incompatible ways in the next release.

StoredFieldsWriter

Codec API for writing stored fields:

  1. For every document, StartDocument(Int32) is called, informing the Codec how many fields will be written.
  2. WriteField(FieldInfo, IIndexableField) is called for each field in the document.
  3. After all documents have been written, Finish(FieldInfos, Int32) is called for verification/sanity-checks.
  4. Finally the writer is disposed (Dispose(Boolean))

Note

This API is experimental and might change in incompatible ways in the next release.

TermsConsumer

Abstract API that consumes terms for an individual field.

The lifecycle is:

  1. TermsConsumer is returned for each field by AddField(FieldInfo).
  2. TermsConsumer returns a PostingsConsumer for each term in StartTerm(BytesRef).
  3. When the producer (e.g. IndexWriter) is done adding documents for the term, it calls FinishTerm(BytesRef, TermStats), passing in the accumulated term statistics.
  4. Producer calls Finish(Int64, Int64, Int32) with the accumulated collection statistics when it is finished adding terms to the field.

Note

This API is experimental and might change in incompatible ways in the next release.

TermStats

Holder for per-term statistics.

TermVectorsFormat

Controls the format of term vectors.

TermVectorsReader

Codec API for reading term vectors:

Note

This API is experimental and might change in incompatible ways in the next release.

TermVectorsWriter

Codec API for writing term vectors:

  1. For every document, StartDocument(Int32) is called, informing the Codec how many fields will be written.
  2. StartField(FieldInfo, Int32, Boolean, Boolean, Boolean) is called for each field in the document, informing the codec how many terms will be written for that field, and whether or not positions, offsets, or payloads are enabled.
  3. Within each field, StartTerm(BytesRef, Int32) is called for each term.
  4. If offsets and/or positions are enabled, then AddPosition(Int32, Int32, Int32, BytesRef) will be called for each term occurrence.
  5. After all documents have been written, Finish(FieldInfos, Int32) is called for verification/sanity-checks.
  6. Finally the writer is disposed (Dispose(Boolean))

Note

This API is experimental and might change in incompatible ways in the next release.

Interfaces

ICodecFactory

Contract for extending the functionality of Codec implementations so they can be injected with dependencies.

To set the ICodecFactory, call SetCodecFactory(ICodecFactory).

See the Lucene.Net.Codecs namespace documentation for some common usage examples.

IDocValuesFormatFactory

Contract for extending the functionality of DocValuesFormat implementations so they can be injected with dependencies.

To set the IDocValuesFormatFactory, call SetDocValuesFormatFactory(IDocValuesFormatFactory).

See the Lucene.Net.Codecs namespace documentation for some common usage examples.

IPostingsFormatFactory

Contract for extending the functionality of PostingsFormat implementations so they can be injected with dependencies.

To set the IPostingsFormatFactory, call SetPostingsFormatFactory(IPostingsFormatFactory).

See the Lucene.Net.Codecs namespace documentation for some common usage examples.