Show / Hide Table of Contents

    Class MockTokenizer

    Tokenizer for testing.

    This tokenizer is a replacement for WHITESPACE, SIMPLE, and KEYWORD tokenizers. If you are writing a component such as a , its a great idea to test it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:

    • An internal state-machine is used for checking consumer consistency. These checks can be disabled with EnableChecks.
    • For convenience, optionally lowercases terms that it outputs.

    Inheritance
    System.Object
    MockTokenizer
    Namespace: Lucene.Net.Analysis
    Assembly: Lucene.Net.TestFramework.dll
    Syntax
    public class MockTokenizer : Tokenizer

    Constructors

    | Improve this Doc View Source

    MockTokenizer(AttributeFactory, TextReader)

    Calls MockTokenizer(AttributeFactory, TextReader, WHITESPACE, true)

    Declaration
    public MockTokenizer(AttributeFactory factory, TextReader input)
    Parameters
    Type Name Description
    AttributeFactory factory
    TextReader input
    | Improve this Doc View Source

    MockTokenizer(AttributeFactory, TextReader, CharacterRunAutomaton, Boolean)

    Declaration
    public MockTokenizer(AttributeFactory factory, TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase)
    Parameters
    Type Name Description
    AttributeFactory factory
    TextReader input
    CharacterRunAutomaton runAutomaton
    System.Boolean lowerCase
    | Improve this Doc View Source

    MockTokenizer(AttributeFactory, TextReader, CharacterRunAutomaton, Boolean, Int32)

    Declaration
    public MockTokenizer(AttributeFactory factory, TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase, int maxTokenLength)
    Parameters
    Type Name Description
    AttributeFactory factory
    TextReader input
    CharacterRunAutomaton runAutomaton
    System.Boolean lowerCase
    System.Int32 maxTokenLength
    | Improve this Doc View Source

    MockTokenizer(TextReader)

    Calls MockTokenizer(TextReader, WHITESPACE, true).

    Declaration
    public MockTokenizer(TextReader input)
    Parameters
    Type Name Description
    TextReader input
    | Improve this Doc View Source

    MockTokenizer(TextReader, CharacterRunAutomaton, Boolean)

    Declaration
    public MockTokenizer(TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase)
    Parameters
    Type Name Description
    TextReader input
    CharacterRunAutomaton runAutomaton
    System.Boolean lowerCase
    | Improve this Doc View Source

    MockTokenizer(TextReader, CharacterRunAutomaton, Boolean, Int32)

    Declaration
    public MockTokenizer(TextReader input, CharacterRunAutomaton runAutomaton, bool lowerCase, int maxTokenLength)
    Parameters
    Type Name Description
    TextReader input
    CharacterRunAutomaton runAutomaton
    System.Boolean lowerCase
    System.Int32 maxTokenLength

    Fields

    | Improve this Doc View Source

    DEFAULT_MAX_TOKEN_LENGTH

    Declaration
    public static readonly int DEFAULT_MAX_TOKEN_LENGTH
    Field Value
    Type Description
    System.Int32
    | Improve this Doc View Source

    KEYWORD

    Acts Similar to KeywordTokenizer. TODO: Keyword returns an "empty" token for an empty reader...

    Declaration
    public static readonly CharacterRunAutomaton KEYWORD
    Field Value
    Type Description
    CharacterRunAutomaton
    | Improve this Doc View Source

    SIMPLE

    Acts like LetterTokenizer.

    Declaration
    public static readonly CharacterRunAutomaton SIMPLE
    Field Value
    Type Description
    CharacterRunAutomaton
    | Improve this Doc View Source

    WHITESPACE

    Acts Similar to WhitespaceTokenizer.

    Declaration
    public static readonly CharacterRunAutomaton WHITESPACE
    Field Value
    Type Description
    CharacterRunAutomaton

    Properties

    | Improve this Doc View Source

    EnableChecks

    Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.

    Declaration
    public virtual bool EnableChecks { get; set; }
    Property Value
    Type Description
    System.Boolean

    Methods

    | Improve this Doc View Source

    Dispose(Boolean)

    Declaration
    protected override void Dispose(bool disposing)
    Parameters
    Type Name Description
    System.Boolean disposing
    | Improve this Doc View Source

    End()

    Declaration
    public override void End()
    | Improve this Doc View Source

    IncrementToken()

    Declaration
    public override sealed bool IncrementToken()
    Returns
    Type Description
    System.Boolean
    | Improve this Doc View Source

    IsTokenChar(Int32)

    Declaration
    protected virtual bool IsTokenChar(int c)
    Parameters
    Type Name Description
    System.Int32 c
    Returns
    Type Description
    System.Boolean
    | Improve this Doc View Source

    Normalize(Int32)

    Declaration
    protected virtual int Normalize(int c)
    Parameters
    Type Name Description
    System.Int32 c
    Returns
    Type Description
    System.Int32
    | Improve this Doc View Source

    ReadChar()

    Declaration
    protected virtual int ReadChar()
    Returns
    Type Description
    System.Int32
    | Improve this Doc View Source

    ReadCodePoint()

    Declaration
    protected virtual int ReadCodePoint()
    Returns
    Type Description
    System.Int32
    | Improve this Doc View Source

    Reset()

    Declaration
    public override void Reset()
    • Improve this Doc
    • View Source
    Back to top Copyright © 2020 Licensed to the Apache Software Foundation (ASF)