Class OpenNLPTokenizer

Run OpenNLP SentenceDetector and Tokenizer. The last token in each sentence is marked by setting the EOS_FLAG_BIT in the IFlagsAttribute; following filters can use this information to apply operations to tokens one sentence at a time.

Inheritance
System.Object
AttributeSource
TokenStream
Tokenizer
SegmentingTokenizerBase
OpenNLPTokenizer
Implements
IDisposable
Inherited Members
Namespace: Lucene.Net.Analysis.OpenNlp
Assembly: Lucene.Net.Analysis.OpenNLP.dll
Syntax
public sealed class OpenNLPTokenizer : SegmentingTokenizerBase, IDisposable

Constructors

| Improve this Doc View Source

OpenNLPTokenizer(AttributeSource.AttributeFactory, TextReader, NLPSentenceDetectorOp, NLPTokenizerOp)

Declaration
public OpenNLPTokenizer(AttributeSource.AttributeFactory factory, TextReader reader, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp)
Parameters
Type Name Description
AttributeSource.AttributeFactory factory
TextReader reader
NLPSentenceDetectorOp sentenceOp
NLPTokenizerOp tokenizerOp
| Improve this Doc View Source

OpenNLPTokenizer(TextReader, NLPSentenceDetectorOp, NLPTokenizerOp)

Creates a new OpenNLPTokenizer

Declaration
public OpenNLPTokenizer(TextReader reader, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp)
Parameters
Type Name Description
TextReader reader
NLPSentenceDetectorOp sentenceOp
NLPTokenizerOp tokenizerOp

Fields

| Improve this Doc View Source

EOS_FLAG_BIT

Declaration
public static int EOS_FLAG_BIT
Field Value
Type Description
System.Int32

Methods

| Improve this Doc View Source

Dispose(Boolean)

Declaration
protected override void Dispose(bool disposing)
Parameters
Type Name Description
System.Boolean disposing
| Improve this Doc View Source

IncrementWord()

Declaration
protected override bool IncrementWord()
Returns
Type Description
System.Boolean
Overrides
| Improve this Doc View Source

Reset()

Declaration
public override void Reset()
Overrides
| Improve this Doc View Source

SetNextSentence(Int32, Int32)

Declaration
protected override void SetNextSentence(int sentenceStart, int sentenceEnd)
Parameters
Type Name Description
System.Int32 sentenceStart
System.Int32 sentenceEnd
Overrides

Implements

IDisposable