kakao/khaiii

What it solves

khaiii is a Korean morphological analyzer designed to split words into their smallest meaningful units (morphemes). This is a fundamental preprocessing step for natural language processing (NLP) tasks such as syntactic and semantic analysis of the Korean language.

How it works

Unlike previous versions that relied on dictionaries and rules, khaiii uses a data-driven approach based on machine learning. It employs a Convolutional Neural Network (CNN) instead of Recurrent Neural Networks (RNNs/LSTMs) to prioritize processing speed, as morphological analysis is a critical preprocessing step.

To handle errors and the "black box" nature of neural networks, the project provides two user dictionary mechanisms:

• Pre-analyzed Dictionary: For words that always have the same analysis result regardless of context (e.g., specific terminology).
• Misanalysis Patch: For correcting specific incorrect analyses across multiple words with context.

Who it’s for

Developers and researchers working on Korean natural language processing (NLP) who need a fast, machine-learning-based morphological analyzer for text preprocessing.

Highlights

• CNN-based architecture: Optimized for speed over RNN-based alternatives.
• Data-driven: Trained on a refined version of the 21st Century Sejong Project corpus (approx. 850,000 sentences).
• Customizable: Includes mechanisms to override neural network results via pre-analyzed dictionaries and patches.
• Multiple model sizes: Offers "base" and "large" models to balance accuracy (F-Score) and processing speed.