bab2min/Kiwi

Kiwi(지능형 한국어 형태소 분석기)

What it solves

Kiwi is a high-performance Korean morphological analyzer designed to break down Korean text into its constituent morphemes (the smallest units of meaning). It specifically addresses the challenge of ambiguity in the Korean language, where different combinations of morphemes can result in the same written form, and provides a way to handle common typos automatically.

How it works

Built with a C++ core for speed, Kiwi combines a statistical language model for local context and a Skip-Bigram model for global context to resolve ambiguity. It is trained on the Sejong and Modu corpora and follows the Sejong part-of-speech (POS) tag set. To ensure accessibility, the core library is wrapped for numerous languages including Python, Java, C#, Go, Rust, and WebAssembly.

Who it’s for

It is intended for developers and researchers working in natural language processing (NLP), text mining, and digital humanities who need a fast, accurate, and easy-to-integrate tool for analyzing Korean text.

Highlights

  • High Speed: Optimized for fast text analysis and supports multi-threading for processing large datasets.
  • Ambiguity Resolution: Uses a lightweight language model to accurately distinguish between similar-looking morpheme combinations.
  • Typo Correction: Includes built-in functionality to automatically correct simple typos during analysis.
  • Broad Integration: Offers wrappers for a wide variety of programming languages and a standalone GUI for non-programmers.

Related

  • Project
    bab2min/kiwipiepy

    A Korean morphological analyzer for Python that provides tokenization, part-of-speech tagging, and advanced typo and dialect correction.

  • Project
    kakao/khaiii

    A CNN-based Korean morphological analyzer developed by Kakao that uses machine learning to split text into morphemes for NLP preprocessing.

  • Project
    epoko77-ai/im-not-ai

    A Claude Code and Codex CLI skill that removes AI-typical patterns from Korean text using a research-based taxonomy to make it sound natural without altering the original meaning.

  • Project
    ikawaha/kagome

    A pure Go Japanese morphological analyzer that tokenizes text and analyzes parts of speech with embedded dictionaries for easy deployment.

  • Project
    djfksjd/ir-search

    A plugin for AI agents that exhaustively crawls and analyzes South Korean government support programs to match startups with funding and grants based on their specific profile.