Class OpenNLPTokenizer

Run OpenNLP SentenceDetector and Tokenizer. The last token in each sentence is marked by setting the EOS_FLAG_BIT in the IFlagsAttribute; following filters can use this information to apply operations to tokens one sentence at a time.

Inheritance

System.Object

AttributeSource

TokenStream

Tokenizer

SegmentingTokenizerBase

OpenNLPTokenizer

Implements

IDisposable

Inherited Members

SegmentingTokenizerBase.BUFFERMAX

SegmentingTokenizerBase.m_buffer

SegmentingTokenizerBase.m_offset

SegmentingTokenizerBase.IncrementToken()

SegmentingTokenizerBase.End()

SegmentingTokenizerBase.IsSafeEnd(Char)

Tokenizer.m_input

Tokenizer.CorrectOffset(Int32)

Tokenizer.SetReader(TextReader)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

Namespace: Lucene.Net.Analysis.OpenNlp

Assembly: Lucene.Net.Analysis.OpenNLP.dll

Syntax

public sealed class OpenNLPTokenizer : SegmentingTokenizerBase, IDisposable

Constructors

OpenNLPTokenizer(AttributeSource.AttributeFactory, TextReader, NLPSentenceDetectorOp, NLPTokenizerOp)

Declaration

public OpenNLPTokenizer(AttributeSource.AttributeFactory factory, TextReader reader, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp)

Parameters

Type	Name	Description
AttributeSource.AttributeFactory	factory
TextReader	reader
NLPSentenceDetectorOp	sentenceOp
NLPTokenizerOp	tokenizerOp

OpenNLPTokenizer(TextReader, NLPSentenceDetectorOp, NLPTokenizerOp)

Creates a new OpenNLPTokenizer

Declaration

public OpenNLPTokenizer(TextReader reader, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp)

Parameters

Type	Name	Description
TextReader	reader
NLPSentenceDetectorOp	sentenceOp
NLPTokenizerOp	tokenizerOp

Fields

EOS_FLAG_BIT

Declaration

public static int EOS_FLAG_BIT

Field Value

Type	Description
System.Int32

Methods

Dispose(Boolean)

Declaration

protected override void Dispose(bool disposing)

Parameters

Type	Name	Description
System.Boolean	disposing

IncrementWord()

Declaration

protected override bool IncrementWord()

Returns

Type	Description
System.Boolean

Overrides

SegmentingTokenizerBase.IncrementWord()

Reset()

Declaration

public override void Reset()

Overrides

SegmentingTokenizerBase.Reset()

SetNextSentence(Int32, Int32)

Declaration

protected override void SetNextSentence(int sentenceStart, int sentenceEnd)

Parameters

Type	Name	Description
System.Int32	sentenceStart
System.Int32	sentenceEnd

Overrides

SegmentingTokenizerBase.SetNextSentence(Int32, Int32)

Implements

IDisposable