Namespace Lucene.Net.Analysis.Compound.Hyphenation

The code for the compound word hyphenation is taken from the Apache FOP project. All credits for the hyphenation code belongs to them.

Classes

ByteVector

This class implements a simple byte vector with access to the underlying array. This class has been taken from the Apache FOP project (http://xmlgraphics.apache.org/fop/). They have been slightly modified.

CharVector

This class implements a simple char vector with access to the underlying array.

This class has been taken from the Apache FOP project (http://xmlgraphics.apache.org/fop/). They have been slightly modified.

Hyphen

This class represents a hyphen. A 'full' hyphen is made of 3 parts: the pre-break text, post-break text and no-break. If no line-break is generated at this position, the no-break text is used, otherwise, pre-break and post-break are used. Typically, pre-break is equal to the hyphen character and the others are empty. However, this general scheme allows support for cases in some languages where words change spelling if they're split across lines, like german's 'backen' which hyphenates 'bak-ken'. BTW, this comes from TeX.

This class has been taken from the Apache FOP project (http://xmlgraphics.apache.org/fop/). They have been slightly modified.

Hyphenation

This class represents a hyphenated word.

This class has been taken from the Apache FOP project (http://xmlgraphics.apache.org/fop/). They have been slightly modified.

HyphenationTree

This tree structure stores the hyphenation patterns in an efficient way for fast lookup. It provides the provides the method to hyphenate a word.

This class has been taken from the Apache FOP project (http://xmlgraphics.apache.org/fop/). They have been slightly modified.

Lucene.NET specific note: If you are going to extend this class by inheriting from it, you should be aware that the base class TernaryTree initializes its state in the constructor by calling its protected Init() method. If your subclass needs to initialize its own state, you add your own "Initialize()" method and call it both from the inside of your constructor and you will need to override the Balance() method and call "Initialize()" before the call to base.Balance(). Your class can use the data that is initialized in the base class after the call to base.Balance().

PatternParser

A XMLReader document handler to read and parse hyphenation patterns from a XML file.

LUCENENET: This class has been refactored from its Java counterpart to use XmlReader rather than a SAX parser.

TernaryTree

Ternary Search Tree.

A ternary search tree is a hybrid between a binary tree and a digital search tree (trie). Keys are limited to strings. A data value of type char is stored in each leaf node. It can be used as an index (or pointer) to the data. Branches that only contain one key are compressed to one node by storing a pointer to the trailer substring of the key. This class is intended to serve as base class or helper class to implement Dictionary collections or the like. Ternary trees have some nice properties as the following: the tree can be traversed in sorted order, partial matches (wildcard) can be implemented, retrieval of all keys within a given distance from the target, etc. The storage requirements are higher than a binary tree but a lot less than a trie. Performance is comparable with a hash table, sometimes it outperforms a hash function (most of the time can determine a miss faster than a hash).

The main purpose of this java port is to serve as a base for implementing TeX's hyphenation algorithm (see The TeXBook, appendix H). Each language requires from 5000 to 15000 hyphenation patterns which will be keys in this tree. The strings patterns are usually small (from 2 to 5 characters), but each char in the tree is stored in a node. Thus memory usage is the main concern. We will sacrifice 'elegance' to keep memory requirements to the minimum. Using java's char type as pointer (yes, I know pointer it is a forbidden word in java) we can keep the size of the node to be just 8 bytes (3 pointers and the data char). This gives room for about 65000 nodes. In my tests the english patterns took 7694 nodes and the german patterns 10055 nodes, so I think we are safe.

All said, this is a map with strings as keys and char as value. Pretty limited!. It can be extended to a general map by using the string representation of an object and using the char value as an index to an array that contains the object values.

This class has been taken from the Apache FOP project (http://xmlgraphics.apache.org/fop/). They have been slightly modified.

TernaryTree.Enumerator

Enumerator for TernaryTree

LUCENENET NOTE: This differs a bit from its Java counterpart to adhere to .NET IEnumerator semantics. In Java, when the TernaryTree.Enumerator is instantiated, it is already positioned at the first element. However, to act like a .NET IEnumerator, the initial state is undefined and considered to be before the first element until MoveNext() is called, and if a move took place it will return true;

Interfaces

IPatternConsumer

This interface is used to connect the XML pattern file parser to the hyphenation tree.

This interface has been taken from the Apache FOP project (http://xmlgraphics.apache.org/fop/). They have been slightly modified.