bab2min/kiwipiepy

Python API for Kiwi

What it solves

Kiwipiepy is a Korean morphological analyzer that breaks down text into its constituent morphemes (the smallest units of meaning) and assigns part-of-speech (POS) tags to them. It addresses the challenge of processing the complex agglutinative nature of the Korean language, providing tools to handle typos, dialects, and custom vocabulary.

How it works

It provides a Python API for the Kiwi morphological analyzer. The system uses a dictionary-based approach with support for user-defined words and rules. It includes specialized logic for handling Korean-specific linguistic features such as "saisiot" (intercalary 's') and various typo correction modes (basic, continual, and lengthening). It also supports OOV (Out-of-Vocabulary) detection using either rule-based or lightweight character-based models.

Who it’s for

Developers and data scientists working on Natural Language Processing (NLP) for Korean text, including those building search engines, chatbots, or text analysis tools that require precise tokenization and POS tagging.

Highlights

  • Advanced Typo Correction: Supports multiple modes to fix basic typos, combined consonants/vowels (continual), and elongated vowels for emphasis (lengthening).
  • Customizable Dictionary: Allows adding user words, including verbs and adjectives with automatic conjugation handling, and supports words containing spaces.
  • Dialect Support: Includes basic analysis capabilities for various Korean dialects (e.g., Jeju, Gyeongsang, Jeolla) and archaic Korean.
  • Flexible Tokenization: Offers options for handling "saisiot" and compatible Jamo output for better compatibility with other systems.
  • OOV Detection: Uses a character-based model to prevent unknown words from being incorrectly split into meaningless fragments.

Related

  • Project
    bab2min/Kiwi

    A fast and accurate Korean morphological analyzer that uses statistical language models and Skip-Bigrams to resolve linguistic ambiguity and process large volumes of text.

  • Project
    kakao/khaiii

    A CNN-based Korean morphological analyzer developed by Kakao that uses machine learning to split text into morphemes for NLP preprocessing.

  • Project
    epoko77-ai/im-not-ai

    A CLI tool for Claude Code, Copilot, and Codex that removes AI-specific patterns and translationese from Korean text to make it sound natural and human-like without changing the original meaning.

  • Project
    WorksApplications/SudachiDict

    A lexicon for the Sudachi Japanese tokenizer, providing various levels of vocabulary coverage to enable accurate text segmentation and analysis.

  • Project
    shibing624/pycorrector

    pycorrector is a Python toolkit for Chinese text correction. It offers a fast KenLM‑based rule system and a suite of neural models (MacBERT, T5, ERNIE, GPT‑style LLMs, BART, etc.) that can be used out‑of‑the‑box or fine‑tuned. Install via pip, pick a model class, and call `correct()` or `correct_batch()` to get corrected sentences and error details. The repo includes evaluation scripts, demos, and command‑line batch processing, targeting use‑cases like input methods, OCR/ASR post‑processing, and search query cleaning.