ku-nlp/jumanpp

Juman++ (a Morphological Analyzer Toolkit)

What it solves

Juman++ is a morphological analyzer designed for languages that lack explicit word boundaries (like Japanese). It breaks down continuous text into individual words and assigns grammatical information to each, using a recurrent neural network language model (RNNLM) to improve accuracy by evaluating the semantic plausibility of word sequences.

How it works

It processes input text by building a lattice of potential word segmentations and grammatical tags. It then uses an RNN-based language model to score these paths, selecting the most likely sequence. Version 2 of the tool significantly improves analysis speed (over 250x faster than the original) and accuracy by optimizing the underlying inference engine.

Who it’s for

It is primarily intended for researchers and developers working on Natural Language Processing (NLP) tasks for Japanese or other languages with similar structural characteristics. It is a general-purpose toolkit that can be adapted for different languages or specific predictive text applications.

Highlights

  • Uses RNN-based language modeling for high-accuracy morphological analysis.
  • Highly optimized C++ implementation for fast performance.
  • Supports custom dictionary training for specialized domains.
  • Includes tools for analyzing path differences in beam search configurations.
  • Designed as a general toolkit, not strictly limited to Japanese.

Related

  • Project
    ali2000hos/persian-writing

    A professional Persian writing and RTL document toolkit that removes "AI-sounding" prose and fixes formatting errors in AI-generated Farsi text.

  • Project
    vespa-engine/sample-apps

    A collection of reference applications and tutorials for Vespa, demonstrating how to implement hybrid search, RAG, and multimodal vector retrieval at scale.

  • Project
    okineadev/vitepress-plugin-llms

    A VitePress plugin that automatically creates LLM‑friendly Markdown files (`llms.txt` and `llms‑full.txt`) and optional UI buttons for copying/downloading pages as Markdown, with support for custom labels, multilingual configs, and special `<llm-only>` / `<llm-exclude>` tags.

  • Project
    tshatrov/ichiran

    Ichiran is a collection of Common Lisp tools for Japanese text processing, providing sentence segmentation, romanization, and dictionary lookups using the JMdict database.