bab2min/kiwipiepy

Python API for Kiwi

What it solves

Kiwipiepy is a Korean morphological analyzer that breaks down Korean text into its smallest meaningful units (morphemes) and assigns part-of-speech (POS) tags to them. It addresses the complexity of the Korean language, including handling typos, dialects, and the need for custom user dictionaries to ensure accurate analysis of specialized terms.

How it works

It provides a Python API for the Kiwi morphological analyzer. The system allows users to tokenize text, split text into sentences, and reconstruct sentences from morphemes. It features a robust set of tools for refining analysis, such as a Stopwords class for filtering, the ability to add user-defined words or regular expression rules for variations, and a pre-analysis correction tool for fixing specific misanalyses.

Who it’s for

Developers and data scientists working with Korean Natural Language Processing (NLP) who need a reliable, customizable tool for tokenization and part-of-speech tagging.

Highlights

  • Advanced Typo Correction: Supports multiple levels of typo correction, including basic typos, continual (connected) typos, and lengthening errors.
  • Customizable Dictionary: Allows adding user words (including those with spaces) and automatically handles conjugation for added verbs and adjectives.
  • Dialect Support: Includes basic analysis capabilities for various Korean dialects (e.g., Jeju, Gyeongsang, Jeolla) and archaic Korean.
  • Flexible Tokenization: Offers options to handle "saisiot" (intercalary 's'), normalize coda consonants, and detect Out-of-Vocabulary (OOV) words using either rule-based or lightweight character-model approaches.
  • Multilingual Support: Enhanced handling of foreign characters, emojis, and Unicode scripts.

Related

  • Project
    bab2min/Kiwi

    A fast and accurate Korean morphological analyzer that uses statistical language models and Skip-Bigrams to resolve linguistic ambiguity and correct typos.

  • Project
    kakao/khaiii

    A CNN-based Korean morphological analyzer developed by Kakao that uses machine learning to split text into morphemes for NLP preprocessing.

  • Project
    epoko77-ai/im-not-ai

    A Claude Code and Codex CLI skill that removes AI-typical patterns from Korean text using a research-based taxonomy to make it sound natural without altering the original meaning.

  • Project
    shibing624/pycorrector

    pycorrector is a Python toolkit for Chinese text correction. It offers a fast KenLM‑based rule system and a suite of neural models (MacBERT, T5, ERNIE, GPT‑style LLMs, BART, etc.) that can be used out‑of‑the‑box or fine‑tuned. Install via pip, pick a model class, and call `correct()` or `correct_batch()` to get corrected sentences and error details. The repo includes evaluation scripts, demos, and command‑line batch processing, targeting use‑cases like input methods, OCR/ASR post‑processing, and search query cleaning.

  • Project
    dongrixinyu/JioNLP

    A Python library for Chinese NLP preprocessing and parsing, providing tools for data cleaning, entity extraction, and automatic LLM evaluation.