megagonlabs/ginza

What it solves

GiNZA is an open-source Japanese Natural Language Processing (NLP) library designed to provide high-accuracy linguistic analysis for the Japanese language, specifically following the Universal Dependencies (UD) framework.

How it works

It is built on top of the spaCy framework and integrates several specialized tools for Japanese text processing:

• Tokenization and POS Tagging: Uses SudachiPy for high-accuracy tokenization and part-of-speech tagging.
• Models: Offers multiple model options, including a standard speed-oriented model (ja_ginza) and a high-accuracy Transformer-based model (ja_ginza_electra) trained using Hugging Face Transformers and pretrained on the mC4 dataset.
• Parsing: The parsing model is trained on the UD Japanese BCCWJ dataset.
• Named Entity Recognition (NER): The NER model is trained on the GSK2014-A (2019) BCCWJ edition, supporting both Sekine's Extended Named Entity Hierarchy and extended OntoNotes5.

Who it’s for

Developers and researchers who need professional-grade Japanese NLP capabilities, such as dependency parsing, tokenization, and named entity recognition, within a Python environment.

Highlights

• Universal Dependencies: Based on the UD framework for consistent syntactic annotation.
• Flexible Models: Choice between fast standard models and high-performance Transformer models.
• Command Line Tools: Includes ginza for full parsing (with JSON and CaboCha-style outputs) and ginzame for MeCab-like tokenization.
• GPU Acceleration: Supports CUDA for accelerating Transformer-based models and thinc-apple-ops for Apple Silicon (M1/M2) acceleration.
• Extensive Integration: Seamlessly integrates with the spaCy ecosystem.