Class PackedInt32s

Simplistic compression for array of unsigned long values. Each value is >= 0 and <= a specified maximum value. The values are stored as packed ints, with each value consuming a fixed number of bits.

NOTE: This was PackedInts in Lucene.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Inheritance

System.Object

PackedInt32s

Inherited Members

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Namespace: Lucene.Net.Util.Packed

Assembly: Lucene.Net.dll

Syntax

public static class PackedInt32s

Fields

CODEC_NAME

Declaration

public static string CODEC_NAME

Field Value

Type	Description
System.String

COMPACT

No memory overhead at all, but the returned implementation may be slow.

Declaration

public static float COMPACT

Field Value

Type	Description
System.Single

DEFAULT

At most 20% memory overhead.

Declaration

public static float DEFAULT

Field Value

Type	Description
System.Single

DEFAULT_BUFFER_SIZE

Default amount of memory to use for bulk operations.

Declaration

public static int DEFAULT_BUFFER_SIZE

Field Value

Type	Description
System.Int32

FAST

At most 50% memory overhead, always select a reasonably fast implementation.

Declaration

public static float FAST

Field Value

Type	Description
System.Single

FASTEST

At most 700% memory overhead, always select a direct implementation.

Declaration

public static float FASTEST

Field Value

Type	Description
System.Single

VERSION_BYTE_ALIGNED

Declaration

public static int VERSION_BYTE_ALIGNED

Field Value

Type	Description
System.Int32

VERSION_CURRENT

Declaration

public static int VERSION_CURRENT

Field Value

Type	Description
System.Int32

VERSION_START

Declaration

public static int VERSION_START

Field Value

Type	Description
System.Int32

Methods

BitsRequired(Int64)

Returns how many bits are required to hold values up to and including maxValue.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static int BitsRequired(long maxValue)

Parameters

Type	Name	Description
System.Int64	maxValue	The maximum value that should be representable.

Returns

Type	Description
System.Int32	The amount of bits needed to represent values from 0 to maxValue.

CheckVersion(Int32)

Check the validity of a version number.

Declaration

public static void CheckVersion(int version)

Parameters

Type	Name	Description
System.Int32	version

Copy(PackedInt32s.Reader, Int32, PackedInt32s.Mutable, Int32, Int32, Int32)

Copy src[srcPos:srcPos+len] into dest[destPos:destPos+len] using at most mem bytes.

Declaration

public static void Copy(PackedInt32s.Reader src, int srcPos, PackedInt32s.Mutable dest, int destPos, int len, int mem)

Parameters

Type	Name	Description
PackedInt32s.Reader	src
System.Int32	srcPos
PackedInt32s.Mutable	dest
System.Int32	destPos
System.Int32	len
System.Int32	mem

FastestFormatAndBits(Int32, Int32, Single)

Try to find the PackedInt32s.Format and number of bits per value that would restore from disk the fastest reader whose overhead is less than acceptableOverheadRatio.

The acceptableOverheadRatio parameter makes sense for random-access PackedInt32s.Readers. In case you only plan to perform sequential access on this stream later on, you should probably use COMPACT.

If you don't know how many values you are going to write, use valueCount = -1.

Declaration

public static PackedInt32s.FormatAndBits FastestFormatAndBits(int valueCount, int bitsPerValue, float acceptableOverheadRatio)

Parameters

Type	Name	Description
System.Int32	valueCount
System.Int32	bitsPerValue
System.Single	acceptableOverheadRatio

Returns

Type	Description
PackedInt32s.FormatAndBits

GetDecoder(PackedInt32s.Format, Int32, Int32)

Get a PackedInt32s.IDecoder.

Declaration

public static PackedInt32s.IDecoder GetDecoder(PackedInt32s.Format format, int version, int bitsPerValue)

Parameters

Type	Name	Description
PackedInt32s.Format	format	The format used to store packed System.Int32s.
System.Int32	version	The compatibility version.
System.Int32	bitsPerValue	The number of bits per value.

Returns

Type	Description
PackedInt32s.IDecoder	A decoder.

GetDirectReader(IndexInput)

Construct a direct PackedInt32s.Reader from an IndexInput. this method is useful to restore data from streams which have been created using GetWriter(DataOutput, Int32, Int32, Single).

The returned reader will have very little memory overhead, but every call to Get(Int32) is likely to perform a disk seek.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Reader GetDirectReader(IndexInput in)

Parameters

Type	Name	Description
IndexInput	in	The stream to read data from.

Returns

Type	Description
PackedInt32s.Reader	A direct PackedInt32s.Reader.

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error.

GetDirectReaderNoHeader(IndexInput, PackedInt32s.Format, Int32, Int32, Int32)

Expert: Construct a direct PackedInt32s.Reader from a stream without reading metadata at the beginning of the stream. This method is useful to restore data from streams which have been created using GetWriterNoHeader(DataOutput, PackedInt32s.Format, Int32, Int32, Int32).

The returned reader will have very little memory overhead, but every call to Get(Int32) is likely to perform a disk seek.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Reader GetDirectReaderNoHeader(IndexInput in, PackedInt32s.Format format, int version, int valueCount, int bitsPerValue)

Parameters

Type	Name	Description
IndexInput	in	The stream to read data from.
PackedInt32s.Format	format	The format used to serialize.
System.Int32	version	The version used to serialize the data.
System.Int32	valueCount	How many values the stream holds.
System.Int32	bitsPerValue	The number of bits per value.

Returns

Type	Description
PackedInt32s.Reader	A direct PackedInt32s.Reader.

GetDirectReaderNoHeader(IndexInput, PackedInt32s.Header)

Expert: Construct a direct PackedInt32s.Reader from an IndexInput without reading metadata at the beginning of the stream. this method is useful to restore data when metadata has been previously read using ReadHeader(DataInput).

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Reader GetDirectReaderNoHeader(IndexInput in, PackedInt32s.Header header)

Parameters

Type	Name	Description
IndexInput	in	The stream to read data from, positioned at the beginning of the packed values.
PackedInt32s.Header	header	Metadata result from ReadHeader(DataInput).

Returns

Type	Description
PackedInt32s.Reader	A PackedInt32s.Reader.

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error.

See Also

ReadHeader(DataInput)

GetEncoder(PackedInt32s.Format, Int32, Int32)

Get an PackedInt32s.IEncoder.

Declaration

public static PackedInt32s.IEncoder GetEncoder(PackedInt32s.Format format, int version, int bitsPerValue)

Parameters

Type	Name	Description
PackedInt32s.Format	format	The format used to store packed System.Int32s.
System.Int32	version	The compatibility version.
System.Int32	bitsPerValue	The number of bits per value.

Returns

Type	Description
PackedInt32s.IEncoder	An encoder.

GetMutable(Int32, Int32, PackedInt32s.Format)

Same as GetMutable(Int32, Int32, Single) with a pre-computed number of bits per value and format.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Mutable GetMutable(int valueCount, int bitsPerValue, PackedInt32s.Format format)

Parameters

Type	Name	Description
System.Int32	valueCount
System.Int32	bitsPerValue
PackedInt32s.Format	format

Returns

Type	Description
PackedInt32s.Mutable

GetMutable(Int32, Int32, Single)

Create a packed integer array with the given amount of values initialized to 0. The valueCount and the bitsPerValue cannot be changed after creation. All Mutables known by this factory are kept fully in RAM.

Positive values of acceptableOverheadRatio will trade space for speed by selecting a faster but potentially less memory-efficient implementation. An acceptableOverheadRatio of COMPACT will make sure that the most memory-efficient implementation is selected whereas FASTEST will make sure that the fastest implementation is selected.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Mutable GetMutable(int valueCount, int bitsPerValue, float acceptableOverheadRatio)

Parameters

Type	Name	Description
System.Int32	valueCount	The number of elements.
System.Int32	bitsPerValue	The number of bits available for any given value.
System.Single	acceptableOverheadRatio	An acceptable overhead ratio per value.

Returns

Type	Description
PackedInt32s.Mutable	A mutable packed integer array.

GetReader(DataInput)

Restore a PackedInt32s.Reader from a stream.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Reader GetReader(DataInput in)

Parameters

Type	Name	Description
DataInput	in	The stream to read data from.

Returns

Type	Description
PackedInt32s.Reader	A PackedInt32s.Reader.

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error

GetReaderIterator(DataInput, Int32)

Retrieve PackedInt32s as a PackedInt32s.IReaderIterator.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.IReaderIterator GetReaderIterator(DataInput in, int mem)

Parameters

Type	Name	Description
DataInput	in	Positioned at the beginning of a stored packed int structure.
System.Int32	mem	How much memory the iterator is allowed to use to read-ahead (likely to speed up iteration).

Returns

Type	Description
PackedInt32s.IReaderIterator	An iterator to access the values.

Exceptions

Type	Condition
System.IO.IOException	If the structure could not be retrieved.

GetReaderIteratorNoHeader(DataInput, PackedInt32s.Format, Int32, Int32, Int32, Int32)

Expert: Restore a PackedInt32s.IReaderIterator from a stream without reading metadata at the beginning of the stream. This method is useful to restore data from streams which have been created using GetWriterNoHeader(DataOutput, PackedInt32s.Format, Int32, Int32, Int32).

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.IReaderIterator GetReaderIteratorNoHeader(DataInput in, PackedInt32s.Format format, int version, int valueCount, int bitsPerValue, int mem)

Parameters

Type	Name	Description
DataInput	in	The stream to read data from, positioned at the beginning of the packed values.
PackedInt32s.Format	format	The format used to serialize.
System.Int32	version	The version used to serialize the data.
System.Int32	valueCount	How many values the stream holds.
System.Int32	bitsPerValue	the number of bits per value.
System.Int32	mem	How much memory the iterator is allowed to use to read-ahead (likely to speed up iteration).

Returns

Type	Description
PackedInt32s.IReaderIterator	A PackedInt32s.IReaderIterator.

See Also

GetWriterNoHeader(DataOutput, PackedInt32s.Format, Int32, Int32, Int32)

GetReaderNoHeader(DataInput, PackedInt32s.Format, Int32, Int32, Int32)

Expert: Restore a PackedInt32s.Reader from a stream without reading metadata at the beginning of the stream. This method is useful to restore data from streams which have been created using GetWriterNoHeader(DataOutput, PackedInt32s.Format, Int32, Int32, Int32).

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Reader GetReaderNoHeader(DataInput in, PackedInt32s.Format format, int version, int valueCount, int bitsPerValue)

Parameters

Type	Name	Description
DataInput	in	The stream to read data from, positioned at the beginning of the packed values.
PackedInt32s.Format	format	The format used to serialize.
System.Int32	version	The version used to serialize the data.
System.Int32	valueCount	How many values the stream holds.
System.Int32	bitsPerValue	The number of bits per value.

Returns

Type	Description
PackedInt32s.Reader	A PackedInt32s.Reader.

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error.

See Also

GetWriterNoHeader(DataOutput, PackedInt32s.Format, Int32, Int32, Int32)

GetReaderNoHeader(DataInput, PackedInt32s.Header)

Expert: Restore a PackedInt32s.Reader from a stream without reading metadata at the beginning of the stream. this method is useful to restore data when metadata has been previously read using ReadHeader(DataInput).

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Reader GetReaderNoHeader(DataInput in, PackedInt32s.Header header)

Parameters

Type	Name	Description
DataInput	in	The stream to read data from, positioned at the beginning of the packed values.
PackedInt32s.Header	header	Metadata result from ReadHeader(DataInput).

Returns

Type	Description
PackedInt32s.Reader	A PackedInt32s.Reader.

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error.

See Also

ReadHeader(DataInput)

GetWriter(DataOutput, Int32, Int32, Single)

Create a packed integer array writer for the given output, format, value count, and number of bits per value.

The resulting stream will be long-aligned. this means that depending on the format which is used under the hoods, up to 63 bits will be wasted. An easy way to make sure that no space is lost is to always use a valueCount that is a multiple of 64.

This method writes metadata to the stream, so that the resulting stream is sufficient to restore a PackedInt32s.Reader from it. You don't need to track valueCount or bitsPerValue by yourself. In case this is a problem, you should probably look at GetWriterNoHeader(DataOutput, PackedInt32s.Format, Int32, Int32, Int32).

The acceptableOverheadRatio parameter controls how readers that will be restored from this stream trade space for speed by selecting a faster but potentially less memory-efficient implementation. An acceptableOverheadRatio of COMPACT will make sure that the most memory-efficient implementation is selected whereas FASTEST will make sure that the fastest implementation is selected. In case you are only interested in reading this stream sequentially later on, you should probably use COMPACT.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Writer GetWriter(DataOutput out, int valueCount, int bitsPerValue, float acceptableOverheadRatio)

Parameters

Type	Name	Description
DataOutput	out	The data output.
System.Int32	valueCount	The number of values.
System.Int32	bitsPerValue	The number of bits per value.
System.Single	acceptableOverheadRatio	An acceptable overhead ratio per value.

Returns

Type	Description
PackedInt32s.Writer	A PackedInt32s.Writer.

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error.

GetWriterNoHeader(DataOutput, PackedInt32s.Format, Int32, Int32, Int32)

Expert: Create a packed integer array writer for the given output, format, value count, and number of bits per value.

The resulting stream will be long-aligned. this means that depending on the format which is used, up to 63 bits will be wasted. An easy way to make sure that no space is lost is to always use a valueCount that is a multiple of 64.

This method does not write any metadata to the stream, meaning that it is your responsibility to store it somewhere else in order to be able to recover data from the stream later on:

• format (using Id),
• valueCount,
• bitsPerValue,
• VERSION_CURRENT.

It is possible to start writing values without knowing how many of them you are actually going to write. To do this, just pass -1 as valueCount. On the other hand, for any positive value of valueCount, the returned writer will make sure that you don't write more values than expected and pad the end of stream with zeros in case you have written less than valueCount when calling Finish().

The mem parameter lets you control how much memory can be used to buffer changes in memory before flushing to disk. High values of mem are likely to improve throughput. On the other hand, if speed is not that important to you, a value of 0 will use as little memory as possible and should already offer reasonable throughput.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static PackedInt32s.Writer GetWriterNoHeader(DataOutput out, PackedInt32s.Format format, int valueCount, int bitsPerValue, int mem)

Parameters

Type	Name	Description
DataOutput	out	The data output.
PackedInt32s.Format	format	The format to use to serialize the values.
System.Int32	valueCount	The number of values.
System.Int32	bitsPerValue	The number of bits per value.
System.Int32	mem	How much memory (in bytes) can be used to speed up serialization.

Returns

Type	Description
PackedInt32s.Writer	A PackedInt32s.Writer.

See Also

GetReaderIteratorNoHeader(DataInput, PackedInt32s.Format, Int32, Int32, Int32, Int32)

GetReaderNoHeader(DataInput, PackedInt32s.Format, Int32, Int32, Int32)

MaxValue(Int32)

Calculates the maximum unsigned long that can be expressed with the given number of bits.

Note

This API is for internal purposes only and might change in incompatible ways in the next release.

Declaration

public static long MaxValue(int bitsPerValue)

Parameters

Type	Name	Description
System.Int32	bitsPerValue	The number of bits available for any given value.

Returns

Type	Description
System.Int64	The maximum value for the given bits.

ReadHeader(DataInput)

Expert: reads only the metadata from a stream. This is useful to later restore a stream or open a direct reader via GetReaderNoHeader(DataInput, PackedInt32s.Header) or GetDirectReaderNoHeader(IndexInput, PackedInt32s.Header).

Declaration

public static PackedInt32s.Header ReadHeader(DataInput in)

Parameters

Type	Name	Description
DataInput	in	The stream to read data.

Returns

Type	Description
PackedInt32s.Header	Packed integer metadata.

Exceptions

Type	Condition
System.IO.IOException	If there is a low-level I/O error.

See Also

GetReaderNoHeader(DataInput, PackedInt32s.Header)

GetDirectReaderNoHeader(IndexInput, PackedInt32s.Header)