Show / Hide Table of Contents

    Class CharTokenizer

    An abstract base class for simple, character-oriented tokenizers.

    You must specify the required LuceneVersion compatibility when creating CharTokenizer:

    • As of 3.1, CharTokenizer uses an int based API to normalize and detect token codepoints. See IsTokenChar(Int32) and Normalize(Int32) for details.

    A new CharTokenizer API has been introduced with Lucene 3.1. This API moved from UTF-16 code units to UTF-32 codepoints to eventually add support for supplementary characters. The old char based API has been deprecated and should be replaced with the int based methods IsTokenChar(Int32) and Normalize(Int32).

    As of Lucene 3.1 each CharTokenizer - constructor expects a LuceneVersion argument. Based on the given LuceneVersion either the new API or a backwards compatibility layer is used at runtime. For LuceneVersion < 3.1 the backwards compatibility layer ensures correct behavior even for indexes build with previous versions of Lucene. If a LuceneVersion >= 3.1 is used CharTokenizer requires the new API to be implemented by the instantiated class. Yet, the old char based API is not required anymore even if backwards compatibility must be preserved. CharTokenizer subclasses implementing the new API are fully backwards compatible if instantiated with LuceneVersion < 3.1.

    Note: If you use a subclass of CharTokenizer with LuceneVersion >= 3.1 on an index build with a version < 3.1, created tokens might not be compatible with the terms in your index.

    Inheritance
    System.Object
    AttributeSource
    TokenStream
    Tokenizer
    CharTokenizer
    LetterTokenizer
    WhitespaceTokenizer
    IndicTokenizer
    RussianLetterTokenizer
    Implements
    IDisposable
    Inherited Members
    Tokenizer.m_input
    Tokenizer.Dispose(Boolean)
    Tokenizer.CorrectOffset(Int32)
    Tokenizer.SetReader(TextReader)
    TokenStream.Dispose()
    AttributeSource.GetAttributeFactory()
    AttributeSource.GetAttributeClassesEnumerator()
    AttributeSource.GetAttributeImplsEnumerator()
    AttributeSource.AddAttributeImpl(Attribute)
    AttributeSource.AddAttribute<T>()
    AttributeSource.HasAttributes
    AttributeSource.HasAttribute<T>()
    AttributeSource.GetAttribute<T>()
    AttributeSource.ClearAttributes()
    AttributeSource.CaptureState()
    AttributeSource.RestoreState(AttributeSource.State)
    AttributeSource.GetHashCode()
    AttributeSource.Equals(Object)
    AttributeSource.ReflectAsString(Boolean)
    AttributeSource.ReflectWith(IAttributeReflector)
    AttributeSource.CloneAttributes()
    AttributeSource.CopyTo(AttributeSource)
    AttributeSource.ToString()
    Namespace: Lucene.Net.Analysis.Util
    Assembly: Lucene.Net.Analysis.Common.dll
    Syntax
    public abstract class CharTokenizer : Tokenizer, IDisposable

    Constructors

    | Improve this Doc View Source

    CharTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader)

    Creates a new CharTokenizer instance

    Declaration
    public CharTokenizer(LuceneVersion matchVersion, AttributeSource.AttributeFactory factory, TextReader input)
    Parameters
    Type Name Description
    LuceneVersion matchVersion

    Lucene version to match

    AttributeSource.AttributeFactory factory

    the attribute factory to use for this Tokenizer

    TextReader input

    the input to split up into tokens

    | Improve this Doc View Source

    CharTokenizer(LuceneVersion, TextReader)

    Creates a new CharTokenizer instance

    Declaration
    public CharTokenizer(LuceneVersion matchVersion, TextReader input)
    Parameters
    Type Name Description
    LuceneVersion matchVersion

    Lucene version to match

    TextReader input

    the input to split up into tokens

    Methods

    | Improve this Doc View Source

    End()

    Declaration
    public override sealed void End()
    Overrides
    TokenStream.End()
    | Improve this Doc View Source

    IncrementToken()

    Declaration
    public override sealed bool IncrementToken()
    Returns
    Type Description
    System.Boolean
    Overrides
    TokenStream.IncrementToken()
    | Improve this Doc View Source

    IsTokenChar(Int32)

    Returns true iff a codepoint should be included in a token. This tokenizer generates as tokens adjacent sequences of codepoints which satisfy this predicate. Codepoints for which this is false are used to define token boundaries and are not included in tokens.

    Declaration
    protected abstract bool IsTokenChar(int c)
    Parameters
    Type Name Description
    System.Int32 c
    Returns
    Type Description
    System.Boolean
    | Improve this Doc View Source

    Normalize(Int32)

    Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens.

    Declaration
    protected virtual int Normalize(int c)
    Parameters
    Type Name Description
    System.Int32 c
    Returns
    Type Description
    System.Int32
    | Improve this Doc View Source

    Reset()

    Declaration
    public override void Reset()
    Overrides
    Tokenizer.Reset()

    Implements

    IDisposable
    • Improve this Doc
    • View Source
    Back to top Copyright © 2020 Licensed to the Apache Software Foundation (ASF)