Class WhitespaceTokenizer
A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.
You must specify the required LuceneVersion compatibility when creating WhitespaceTokenizer:
- As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See IsTokenChar(Int32) and Normalize(Int32) for details.
Implements
IDisposable
Inherited Members
Namespace: Lucene.Net.Analysis.Core
Assembly: Lucene.Net.Analysis.Common.dll
Syntax
public sealed class WhitespaceTokenizer : CharTokenizer, IDisposableConstructors
| Improve this Doc View SourceWhitespaceTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader)
Construct a new WhitespaceTokenizer using a given AttributeSource.AttributeFactory.
Declaration
public WhitespaceTokenizer(LuceneVersion matchVersion, AttributeSource.AttributeFactory factory, TextReader in)Parameters
| Type | Name | Description |
|---|---|---|
| LuceneVersion | matchVersion | LuceneVersion to match |
| AttributeSource.AttributeFactory | factory | the attribute factory to use for this Tokenizer |
| TextReader | in | the input to split up into tokens |
WhitespaceTokenizer(LuceneVersion, TextReader)
Declaration
public WhitespaceTokenizer(LuceneVersion matchVersion, TextReader in)Parameters
| Type | Name | Description |
|---|---|---|
| LuceneVersion | matchVersion | LuceneVersion to match |
| TextReader | in | the input to split up into tokens |
Methods
| Improve this Doc View SourceIsTokenChar(Int32)
Collects only characters which do not satisfy
Declaration
protected override bool IsTokenChar(int c)Parameters
| Type | Name | Description |
|---|---|---|
| System.Int32 | c |
Returns
| Type | Description |
|---|---|
| System.Boolean |
Overrides
Implements
IDisposable