Class TokenFilter

A TokenFilter is a TokenStream whose input is another TokenStream.

This is an abstract class; subclasses must override IncrementToken().

Inheritance

System.Object

AttributeSource

TokenStream

TokenFilter

CachingTokenFilter

Implements

System.IDisposable

Inherited Members

TokenStream.IncrementToken()

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

System.Object.Equals(System.Object, System.Object)

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

Namespace: Lucene.Net.Analysis

Assembly: Lucene.Net.dll

Syntax

public abstract class TokenFilter : TokenStream, IDisposable

Constructors

TokenFilter(TokenStream)

Construct a token stream filtering the given input.

Declaration

protected TokenFilter(TokenStream input)

Parameters

Type	Name	Description
TokenStream	input

Fields

m_input

The source of tokens for this filter.

Declaration

protected readonly TokenStream m_input

Field Value

Type	Description
TokenStream

Methods

Dispose(Boolean)

Releases resources associated with this stream.

If you override this method, always call base.Dispose(disposing), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw System.InvalidOperationException on reuse).

NOTE: The default implementation chains the call to the input TokenStream, so be sure to call base.Dispose(disposing) when overriding this method.

Declaration

protected override void Dispose(bool disposing)

Parameters

Type	Name	Description
System.Boolean	disposing

Overrides

TokenStream.Dispose(Boolean)

End()

This method is called by the consumer after the last token has been consumed, after IncrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.

NOTE: The default implementation chains the call to the input TokenStream, so be sure to call base.End() first when overriding this method.

Declaration

public override void End()

Overrides

TokenStream.End()

Exceptions

Type	Condition
System.IO.IOException	If an I/O error occurs

Reset()

This method is called by a consumer before it begins consumption using IncrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call base.Reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw System.InvalidOperationException on further usage).

Declaration

public override void Reset()

Overrides

TokenStream.Reset()

Remarks

NOTE: The default implementation chains the call to the input TokenStream, so be sure to call base.Reset() when overriding this method.

Implements

System.IDisposable

Extension Methods

Number.IsNumber(Object)

See Also

TokenStream