IN - Successor Variety

This method is based on structural linguistic studies of word and morpheme boundaries that consider the distribution of phonemes. We apply them to large collections to get adequate statistical information, and work with letters instead of phonemes.

Successor Variety of a string is the number of different characters that follow it in words in the collection being considered.

Segmentation Methods

• Cutoff: boundary occurs when successor variety jumps.
• Peak and Plateau: break after character whose value is higher than that before or after it.
• Complete word: lookup segment in word list.
• Entropy: compute change at each position and apply threshold
• Count: frequently occurring prefixes are good stems.