Show / Hide Table of Contents

    Class WhitespaceTokenizer

    A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.

    You must specify the required LuceneVersion compatibility when creating WhitespaceTokenizer:

    • As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See IsTokenChar(Int32) and Normalize(Int32) for details.

    Inheritance
    System.Object
    AttributeSource
    TokenStream
    Tokenizer
    CharTokenizer
    WhitespaceTokenizer
    Implements
    IDisposable
    Inherited Members
    CharTokenizer.Normalize(Int32)
    CharTokenizer.IncrementToken()
    CharTokenizer.End()
    CharTokenizer.Reset()
    Tokenizer.m_input
    Tokenizer.Dispose(Boolean)
    Tokenizer.CorrectOffset(Int32)
    Tokenizer.SetReader(TextReader)
    TokenStream.Dispose()
    AttributeSource.GetAttributeFactory()
    AttributeSource.GetAttributeClassesEnumerator()
    AttributeSource.GetAttributeImplsEnumerator()
    AttributeSource.AddAttributeImpl(Attribute)
    AttributeSource.AddAttribute<T>()
    AttributeSource.HasAttributes
    AttributeSource.HasAttribute<T>()
    AttributeSource.GetAttribute<T>()
    AttributeSource.ClearAttributes()
    AttributeSource.CaptureState()
    AttributeSource.RestoreState(AttributeSource.State)
    AttributeSource.GetHashCode()
    AttributeSource.Equals(Object)
    AttributeSource.ReflectAsString(Boolean)
    AttributeSource.ReflectWith(IAttributeReflector)
    AttributeSource.CloneAttributes()
    AttributeSource.CopyTo(AttributeSource)
    AttributeSource.ToString()
    Namespace: Lucene.Net.Analysis.Core
    Assembly: Lucene.Net.Analysis.Common.dll
    Syntax
    public sealed class WhitespaceTokenizer : CharTokenizer, IDisposable

    Constructors

    | Improve this Doc View Source

    WhitespaceTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader)

    Construct a new WhitespaceTokenizer using a given AttributeSource.AttributeFactory.

    Declaration
    public WhitespaceTokenizer(LuceneVersion matchVersion, AttributeSource.AttributeFactory factory, TextReader in)
    Parameters
    Type Name Description
    LuceneVersion matchVersion

    LuceneVersion to match

    AttributeSource.AttributeFactory factory

    the attribute factory to use for this Tokenizer

    TextReader in

    the input to split up into tokens

    | Improve this Doc View Source

    WhitespaceTokenizer(LuceneVersion, TextReader)

    Declaration
    public WhitespaceTokenizer(LuceneVersion matchVersion, TextReader in)
    Parameters
    Type Name Description
    LuceneVersion matchVersion

    LuceneVersion to match

    TextReader in

    the input to split up into tokens

    Methods

    | Improve this Doc View Source

    IsTokenChar(Int32)

    Collects only characters which do not satisfy .

    Declaration
    protected override bool IsTokenChar(int c)
    Parameters
    Type Name Description
    System.Int32 c
    Returns
    Type Description
    System.Boolean
    Overrides
    CharTokenizer.IsTokenChar(Int32)

    Implements

    IDisposable
    • Improve this Doc
    • View Source
    Back to top Copyright © 2020 Licensed to the Apache Software Foundation (ASF)