Class MockTokenizer

Tokenizer for testing.

This tokenizer is a replacement for WHITESPACE, SIMPLE, and KEYWORD tokenizers. If you are writing a component such as a TokenFilter, its a great idea to test it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:

An internal state-machine is used for checking consumer consistency. These checks can be disabled with EnableChecks.
For convenience, optionally lowercases terms that it outputs.

Inheritance

System.Object

AttributeSource

TokenStream

Tokenizer

MockTokenizer

Implements

System.IDisposable

Inherited Members

Tokenizer.m_input

Tokenizer.CorrectOffset(Int32)

Tokenizer.SetReader(TextReader)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

System.Object.Equals(System.Object, System.Object)

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

Namespace: Lucene.Net.Analysis

Assembly: Lucene.Net.TestFramework.dll

Syntax

public class MockTokenizer : Tokenizer, IDisposable

Constructors

| Improve this Doc View Source

MockTokenizer(AttributeSource.AttributeFactory, TextReader)

Calls MockTokenizer(AttributeFactory, TextReader, WHITESPACE, true)

Declaration

public MockTokenizer(AttributeSource.AttributeFactory factory, TextReader input)

Parameters

Type	Name	Description
AttributeSource.AttributeFactory	factory
System.IO.TextReader	input

| Improve this Doc View Source

MockTokenizer(AttributeSource.AttributeFactory, TextReader, CharacterRunAutomaton, Boolean)

Declaration

public MockTokenizer(AttributeSource.AttributeFactory factory, TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase)

Parameters

Type	Name	Description
AttributeSource.AttributeFactory	factory
System.IO.TextReader	input
CharacterRunAutomaton	runAutomaton
System.Boolean	lowerCase

| Improve this Doc View Source

MockTokenizer(AttributeSource.AttributeFactory, TextReader, CharacterRunAutomaton, Boolean, Int32)

Declaration

public MockTokenizer(AttributeSource.AttributeFactory factory, TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase, int maxTokenLength)

Parameters

Type	Name	Description
AttributeSource.AttributeFactory	factory
System.IO.TextReader	input
CharacterRunAutomaton	runAutomaton
System.Boolean	lowerCase
System.Int32	maxTokenLength

| Improve this Doc View Source

MockTokenizer(TextReader)

Calls MockTokenizer(TextReader, WHITESPACE, true).

Declaration

public MockTokenizer(TextReader input)

Parameters

Type	Name	Description
System.IO.TextReader	input

| Improve this Doc View Source

MockTokenizer(TextReader, CharacterRunAutomaton, Boolean)

Declaration

public MockTokenizer(TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase)

Parameters

Type	Name	Description
System.IO.TextReader	input
CharacterRunAutomaton	runAutomaton
System.Boolean	lowerCase

| Improve this Doc View Source

MockTokenizer(TextReader, CharacterRunAutomaton, Boolean, Int32)

Declaration

public MockTokenizer(TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase, int maxTokenLength)

Parameters

Type	Name	Description
System.IO.TextReader	input
CharacterRunAutomaton	runAutomaton
System.Boolean	lowerCase
System.Int32	maxTokenLength

Fields

| Improve this Doc View Source

DEFAULT_MAX_TOKEN_LENGTH

Declaration

public static readonly int DEFAULT_MAX_TOKEN_LENGTH

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

KEYWORD

Acts Similar to KeywordTokenizer. TODO: Keyword returns an "empty" token for an empty reader...

Declaration

public static readonly CharacterRunAutomaton KEYWORD

Field Value

Type	Description
CharacterRunAutomaton

| Improve this Doc View Source

SIMPLE

Acts like LetterTokenizer.

Declaration

public static readonly CharacterRunAutomaton SIMPLE

Field Value

Type	Description
CharacterRunAutomaton

| Improve this Doc View Source

WHITESPACE

Acts Similar to WhitespaceTokenizer.

Declaration

public static readonly CharacterRunAutomaton WHITESPACE

Field Value

Type	Description
CharacterRunAutomaton

Properties

| Improve this Doc View Source

EnableChecks

Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.

Declaration

public virtual bool EnableChecks { get; set; }

Property Value

Type	Description
System.Boolean

Methods

| Improve this Doc View Source

Dispose(Boolean)

Declaration

protected override void Dispose(bool disposing)

Parameters

Type	Name	Description
System.Boolean	disposing

Overrides

Tokenizer.Dispose(Boolean)

| Improve this Doc View Source

End()

Declaration

public override void End()

Overrides

TokenStream.End()

| Improve this Doc View Source

IncrementToken()

Declaration

public override sealed bool IncrementToken()

Returns

Type	Description
System.Boolean

Overrides

TokenStream.IncrementToken()

| Improve this Doc View Source

IsTokenChar(Int32)

Declaration

protected virtual bool IsTokenChar(int c)

Parameters

Type	Name	Description
System.Int32	c

Returns

Type	Description
System.Boolean

| Improve this Doc View Source

Normalize(Int32)

Declaration

protected virtual int Normalize(int c)

Parameters

Type	Name	Description
System.Int32	c

Returns

Type	Description
System.Int32

| Improve this Doc View Source

ReadChar()

Declaration

protected virtual int ReadChar()

Returns

Type	Description
System.Int32

| Improve this Doc View Source

ReadCodePoint()

Declaration

protected virtual int ReadCodePoint()

Returns

Type	Description
System.Int32

| Improve this Doc View Source

Reset()

Declaration

public override void Reset()

Overrides

Tokenizer.Reset()

Implements

System.IDisposable