Class WhitespaceTokenizer

A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.

You must specify the required LuceneVersion compatibility when creating WhitespaceTokenizer:

• As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See IsTokenChar(Int32) and Normalize(Int32) for details.

Inheritance

System.Object

AttributeSource

TokenStream

Tokenizer

CharTokenizer

WhitespaceTokenizer

Implements

System.IDisposable

Inherited Members

CharTokenizer.Normalize(Int32)

CharTokenizer.IncrementToken()

CharTokenizer.End()

CharTokenizer.Reset()

Tokenizer.m_input

Tokenizer.Dispose(Boolean)

Tokenizer.CorrectOffset(Int32)

Tokenizer.SetReader(TextReader)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

System.Object.Equals(System.Object, System.Object)

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

Namespace: Lucene.Net.Analysis.Core

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

public sealed class WhitespaceTokenizer : CharTokenizer, IDisposable

Constructors

WhitespaceTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader)

Construct a new WhitespaceTokenizer using a given AttributeSource.AttributeFactory.

Declaration

public WhitespaceTokenizer(LuceneVersion matchVersion, AttributeSource.AttributeFactory factory, TextReader in)

Parameters

Type	Name	Description
LuceneVersion	matchVersion	LuceneVersion to match
AttributeSource.AttributeFactory	factory	the attribute factory to use for this Tokenizer
System.IO.TextReader	in	the input to split up into tokens

WhitespaceTokenizer(LuceneVersion, TextReader)

Declaration

public WhitespaceTokenizer(LuceneVersion matchVersion, TextReader in)

Parameters

Type	Name	Description
LuceneVersion	matchVersion	LuceneVersion to match
System.IO.TextReader	in	the input to split up into tokens

Methods

IsTokenChar(Int32)

Collects only characters which do not satisfy System.Char.IsWhiteSpace(System.Char).

Declaration

protected override bool IsTokenChar(int c)

Parameters

Type	Name	Description
System.Int32	c

Returns

Type	Description
System.Boolean

Overrides

CharTokenizer.IsTokenChar(Int32)

Implements

System.IDisposable