Class WikipediaTokenizer

Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.

This is a Lucene.NET EXPERIMENTAL API, use at your own risk

Inheritance

System.Object

AttributeSource

TokenStream

Tokenizer

WikipediaTokenizer

Implements

System.IDisposable

Inherited Members

Tokenizer.m_input

Tokenizer.CorrectOffset(Int32)

Tokenizer.SetReader(TextReader)

TokenStream.Dispose()

AttributeSource.GetAttributeFactory()

AttributeSource.GetAttributeClassesEnumerator()

AttributeSource.GetAttributeImplsEnumerator()

AttributeSource.AddAttributeImpl(Attribute)

AttributeSource.AddAttribute<T>()

AttributeSource.HasAttributes

AttributeSource.HasAttribute<T>()

AttributeSource.GetAttribute<T>()

AttributeSource.ClearAttributes()

AttributeSource.CaptureState()

AttributeSource.RestoreState(AttributeSource.State)

AttributeSource.GetHashCode()

AttributeSource.Equals(Object)

AttributeSource.ReflectAsString(Boolean)

AttributeSource.ReflectWith(IAttributeReflector)

AttributeSource.CloneAttributes()

AttributeSource.CopyTo(AttributeSource)

AttributeSource.ToString()

System.Object.Equals(System.Object, System.Object)

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

Namespace: Lucene.Net.Analysis.Wikipedia

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

public sealed class WikipediaTokenizer : Tokenizer, IDisposable

Constructors

| Improve this Doc View Source

WikipediaTokenizer(AttributeSource.AttributeFactory, TextReader, Int32, ICollection<String>)

Creates a new instance of the WikipediaTokenizer. Attaches the input to a the newly created JFlex scanner. Uses the given AttributeSource.AttributeFactory.

Declaration

public WikipediaTokenizer(AttributeSource.AttributeFactory factory, TextReader input, int tokenOutput, ICollection<string> untokenizedTypes)

Parameters

Type	Name	Description
AttributeSource.AttributeFactory	factory	The AttributeSource.AttributeFactory
System.IO.TextReader	input	The input
System.Int32	tokenOutput	One of TOKENS_ONLY, UNTOKENIZED_ONLY, BOTH
System.Collections.Generic.ICollection<System.String>	untokenizedTypes	Untokenized types

| Improve this Doc View Source

WikipediaTokenizer(TextReader)

Creates a new instance of the WikipediaTokenizer. Attaches the input to a newly created JFlex scanner.

Declaration

public WikipediaTokenizer(TextReader input)

Parameters

Type	Name	Description
System.IO.TextReader	input	The Input System.IO.TextReader

| Improve this Doc View Source

WikipediaTokenizer(TextReader, Int32, ICollection<String>)

Creates a new instance of the WikipediaTokenizer. Attaches the input to a the newly created JFlex scanner.

Declaration

public WikipediaTokenizer(TextReader input, int tokenOutput, ICollection<string> untokenizedTypes)

Parameters

Type	Name	Description
System.IO.TextReader	input	The input
System.Int32	tokenOutput	One of TOKENS_ONLY, UNTOKENIZED_ONLY, BOTH
System.Collections.Generic.ICollection<System.String>	untokenizedTypes	Untokenized types

Fields

| Improve this Doc View Source

ACRONYM_ID

Declaration

public const int ACRONYM_ID = 2

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

ALPHANUM_ID

Declaration

public const int ALPHANUM_ID = 0

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

APOSTROPHE_ID

Declaration

public const int APOSTROPHE_ID = 1

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

BOLD

Declaration

public const string BOLD = "b"

Field Value

Type	Description
System.String

| Improve this Doc View Source

BOLD_ID

Declaration

public const int BOLD_ID = 12

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

BOLD_ITALICS

Declaration

public const string BOLD_ITALICS = "bi"

Field Value

Type	Description
System.String

| Improve this Doc View Source

BOLD_ITALICS_ID

Declaration

public const int BOLD_ITALICS_ID = 14

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

BOTH

Output the both the untokenized token and the splits

Declaration

public const int BOTH = 2

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

CATEGORY_ID

Declaration

public const int CATEGORY_ID = 11

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

CITATION

Declaration

public const string CITATION = "ci"

Field Value

Type	Description
System.String

| Improve this Doc View Source

CITATION_ID

Declaration

public const int CITATION_ID = 10

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

CJ_ID

Declaration

public const int CJ_ID = 7

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

COMPANY_ID

Declaration

public const int COMPANY_ID = 3

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

EMAIL_ID

Declaration

public const int EMAIL_ID = 4

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

EXTERNAL_LINK

Declaration

public const string EXTERNAL_LINK = "el"

Field Value

Type	Description
System.String

| Improve this Doc View Source

EXTERNAL_LINK_ID

Declaration

public const int EXTERNAL_LINK_ID = 9

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

EXTERNAL_LINK_URL

Declaration

public const string EXTERNAL_LINK_URL = "elu"

Field Value

Type	Description
System.String

| Improve this Doc View Source

EXTERNAL_LINK_URL_ID

Declaration

public const int EXTERNAL_LINK_URL_ID = 17

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

HEADING

Declaration

public const string HEADING = "h"

Field Value

Type	Description
System.String

| Improve this Doc View Source

HEADING_ID

Declaration

public const int HEADING_ID = 15

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

HOST_ID

Declaration

public const int HOST_ID = 5

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

INTERNAL_LINK

Declaration

public const string INTERNAL_LINK = "il"

Field Value

Type	Description
System.String

| Improve this Doc View Source

INTERNAL_LINK_ID

Declaration

public const int INTERNAL_LINK_ID = 8

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

ITALICS

Declaration

public const string ITALICS = "i"

Field Value

Type	Description
System.String

| Improve this Doc View Source

ITALICS_ID

Declaration

public const int ITALICS_ID = 13

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

NUM_ID

Declaration

public const int NUM_ID = 6

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

SUB_HEADING

Declaration

public const string SUB_HEADING = "sh"

Field Value

Type	Description
System.String

| Improve this Doc View Source

SUB_HEADING_ID

Declaration

public const int SUB_HEADING_ID = 16

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

TOKEN_TYPES

String token types that correspond to token type int constants

Declaration

public static readonly string[] TOKEN_TYPES

Field Value

Type	Description
System.String[]

| Improve this Doc View Source

TOKENS_ONLY

Only output tokens

Declaration

public const int TOKENS_ONLY = 0

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

UNTOKENIZED_ONLY

Only output untokenized tokens, which are tokens that would normally be split into several tokens

Declaration

public const int UNTOKENIZED_ONLY = 1

Field Value

Type	Description
System.Int32

| Improve this Doc View Source

UNTOKENIZED_TOKEN_FLAG

This flag is used to indicate that the produced "Token" would, if TOKENS_ONLY was used, produce multiple tokens.

Declaration

public const int UNTOKENIZED_TOKEN_FLAG = 1

Field Value

Type	Description
System.Int32

Methods

| Improve this Doc View Source

Dispose(Boolean)

Declaration

protected override void Dispose(bool disposing)

Parameters

Type	Name	Description
System.Boolean	disposing

Overrides

Tokenizer.Dispose(Boolean)

| Improve this Doc View Source

End()

Declaration

public override void End()

Overrides

TokenStream.End()

| Improve this Doc View Source

IncrementToken()

Declaration

public override sealed bool IncrementToken()

Returns

Type	Description
System.Boolean

Overrides

TokenStream.IncrementToken()

| Improve this Doc View Source

Reset()

Declaration

public override void Reset()

Overrides

Tokenizer.Reset()

Implements

System.IDisposable