Class PatternAnalyzer

Efficient Lucene analyzer/tokenizer that preferably operates on a rather than a , that can flexibly separate text into terms via a regular expression (with behaviour similar to ), and that combines the functionality of LetterTokenizer, LowerCaseTokenizer, WhitespaceTokenizer, StopFilter into a single efficient multi-purpose class.

If you are unsure how exactly a regular expression should look like, consider prototyping by simply trying various expressions on some test texts via . Once you are satisfied, give that regex to PatternAnalyzer. Also see Regular Expression Tutorial.

This class can be considerably faster than the "normal" Lucene tokenizers. It can also serve as a building block in a compound Lucene TokenFilter chain. For example as in this stemming example:

PatternAnalyzer pat = ...
TokenStream tokenStream = new SnowballFilter(
    pat.GetTokenStream("content", "James is running round in the woods"), 
    "English"));

Inheritance

System.Object

Analyzer

PatternAnalyzer

Inherited Members

Lucene.Net.Analysis.Analyzer.NewAnonymous(Func<, , >)

Lucene.Net.Analysis.Analyzer.NewAnonymous(Func<, , >, Lucene.Net.Analysis.ReuseStrategy)

Lucene.Net.Analysis.Analyzer.NewAnonymous(Func<, , >, Func<, , >)

Lucene.Net.Analysis.Analyzer.NewAnonymous(Func<, , >, Func<, , >, Lucene.Net.Analysis.ReuseStrategy)

Analyzer.GetTokenStream(String, TextReader)

Analyzer.GetTokenStream(String, String)

Analyzer.InitReader(String, TextReader)

Analyzer.GetPositionIncrementGap(String)

Analyzer.GetOffsetGap(String)

Analyzer.Strategy

Analyzer.Dispose()

Analyzer.Dispose(Boolean)

Analyzer.GLOBAL_REUSE_STRATEGY

Analyzer.PER_FIELD_REUSE_STRATEGY

Namespace: Lucene.Net.Analysis.Miscellaneous

Assembly: Lucene.Net.Analysis.Common.dll

Syntax

public sealed class PatternAnalyzer : Analyzer

Constructors

PatternAnalyzer(LuceneVersion, Regex, Boolean, CharArraySet)

Constructs a new instance with the given parameters.

Declaration

public PatternAnalyzer(LuceneVersion matchVersion, Regex pattern, bool toLowerCase, CharArraySet stopWords)

Parameters

Type	Name	Description
LuceneVersion	matchVersion	currently does nothing
Regex	pattern	a regular expression delimiting tokens
System.Boolean	toLowerCase	if true returns tokens after applying String.toLowerCase()
CharArraySet	stopWords	if non-null, ignores all tokens that are contained in the given stop set (after previously having applied toLowerCase() if applicable). For example, created via MakeStopSet(LuceneVersion, String[])and/or WordlistLoaderas in WordlistLoader.getWordSet(new File("samples/fulltext/stopwords.txt") or other stop words lists .

Fields

DEFAULT_ANALYZER

A lower-casing word analyzer with English stop words (can be shared freely across threads without harm); global per class loader.

Declaration

public static readonly PatternAnalyzer DEFAULT_ANALYZER

Field Value

Type	Description
PatternAnalyzer

EXTENDED_ANALYZER

A lower-casing word analyzer with extended English stop words (can be shared freely across threads without harm); global per class loader. The stop words are borrowed from http://thomas.loc.gov/home/stopwords.html, see http://thomas.loc.gov/home/all.about.inquery.html

Declaration

public static readonly PatternAnalyzer EXTENDED_ANALYZER

Field Value

Type	Description
PatternAnalyzer

NON_WORD_PATTERN

"\W+"; Divides text at non-letters (NOT Character.isLetter(c))

Declaration

public static readonly Regex NON_WORD_PATTERN

Field Value

Type	Description
Regex

WHITESPACE_PATTERN

"\s+"; Divides text at whitespaces (Character.isWhitespace(c))

Declaration

public static readonly Regex WHITESPACE_PATTERN

Field Value

Type	Description
Regex

Methods

CreateComponents(String, TextReader)

Creates a token stream that tokenizes all the text in the given SetReader; This implementation forwards to and is less efficient than .

Declaration

protected override TokenStreamComponents CreateComponents(string fieldName, TextReader reader)

Parameters

Type	Name	Description
System.String	fieldName	the name of the field to tokenize (currently ignored).
TextReader	reader	the reader delivering the text

Returns

Type	Description
TokenStreamComponents	a new token stream

CreateComponents(String, TextReader, String)

Creates a token stream that tokenizes the given string into token terms (aka words).

Declaration

public TokenStreamComponents CreateComponents(string fieldName, TextReader reader, string text)

Parameters

Type	Name	Description
System.String	fieldName	the name of the field to tokenize (currently ignored).
TextReader	reader	reader (e.g. charfilter) of the original text. can be null.
System.String	text	the string to tokenize

Returns

Type	Description
TokenStreamComponents	a new token stream

Equals(Object)

Indicates whether some other object is "equal to" this one.

Declaration

public override bool Equals(object other)

Parameters

Type	Name	Description
System.Object	other	the reference object with which to compare.

Returns

Type	Description
System.Boolean	true if equal, false otherwise

GetHashCode()

Returns a hash code value for the object.

Declaration

public override int GetHashCode()

Returns

Type	Description
System.Int32	the hash code.