Class TokenStreamToAutomaton

Consumes a TokenStream and creates an Automaton where the transition labels are UTF8 bytes (or Unicode code points if unicodeArcs is true) from the ITermToBytesRefAttribute. Between tokens we insert POS_SEP and for holes we insert HOLE.

Note

This API is experimental and might change in incompatible ways in the next release.

Inheritance
System.Object
TokenStreamToAutomaton
Inherited Members
System.Object.Equals(System.Object)
System.Object.Equals(System.Object, System.Object)
System.Object.GetHashCode()
System.Object.GetType()
System.Object.MemberwiseClone()
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.ToString()
Namespace: Lucene.Net.Analysis
Assembly: Lucene.Net.dll
Syntax
public class TokenStreamToAutomaton

Constructors

| Improve this Doc View Source

TokenStreamToAutomaton()

Sole constructor.

Declaration
public TokenStreamToAutomaton()

Fields

| Improve this Doc View Source

HOLE

We add this arc to represent a hole.

Declaration
public const int HOLE = 30
Field Value
Type Description
System.Int32
| Improve this Doc View Source

POS_SEP

We create transition between two adjacent tokens.

Declaration
public const int POS_SEP = 31
Field Value
Type Description
System.Int32

Properties

| Improve this Doc View Source

PreservePositionIncrements

Whether to generate holes in the automaton for missing positions, true by default.

Declaration
public virtual bool PreservePositionIncrements { get; set; }
Property Value
Type Description
System.Boolean
| Improve this Doc View Source

UnicodeArcs

Whether to make transition labels Unicode code points instead of UTF8 bytes, false by default

Declaration
public virtual bool UnicodeArcs { get; set; }
Property Value
Type Description
System.Boolean

Methods

| Improve this Doc View Source

ChangeToken(BytesRef)

Subclass & implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.

Declaration
protected virtual BytesRef ChangeToken(BytesRef in)
Parameters
Type Name Description
BytesRef in
Returns
Type Description
BytesRef
| Improve this Doc View Source

ToAutomaton(TokenStream)

Pulls the graph (including IPositionLengthAttribute from the provided TokenStream, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.

Declaration
public virtual Automaton ToAutomaton(TokenStream in)
Parameters
Type Name Description
TokenStream in
Returns
Type Description
Automaton