hankcs/HanLP

HanLP – Multilingual Natural‑Language‑Processing Toolkit

What it is – HanLP is an open‑source Python library (with Java and Go bindings) that provides production‑ready NLP models for many languages. It ships with both a lightweight RESTful service (a few KB client, no GPU needed) and a full‑featured native Python API that runs on PyTorch or TensorFlow 2.x. The project focuses on speed, accuracy, and easy customization, and it bundles dozens of pretrained models covering tokenization, POS‑tagging, NER, dependency/constituency parsing, semantic role labeling, AMR, coreference, text similarity, summarization, style transfer, sentiment analysis, language detection, and more.

---

Key Features (as described in the README)

Feature	How it’s offered	Highlights
Multilingual support	Models for 130 languages (Chinese, English, Japanese, Russian, French, German, etc.)	Unified API across languages
Joint (multi‑task) models	One model can perform up to 10 tasks simultaneously (e.g., tokenization + POS + NER + SRL + dependency + semantic‑dependency + constituency)	Faster inference, lower memory use
Single‑task models	High‑precision models for each task when you need the best accuracy	Flexible pipeline composition
Two API styles	• RESTful – tiny client, works from any language, no GPU required \n• Native – Python library that loads the heavy models locally (CPU/GPU/TPU)	Choose based on deployment scale
Easy customization	Trie‑based user dictionaries, rule‑based token merging/splitting, ability to drop unwanted tasks to shrink models	Adapt quickly to new domains
Consistent JSON output	All APIs return a JSON‑compatible Document object with fields like tok/fine, pos/ctb, ner/msra, dep, srl, etc.	Simple downstream processing
Visualization	Console‑based tree visualisation for dependency/semantic structures	Helpful for debugging and teaching
Production‑ready	Unit‑tested on Linux/macOS/Windows, supports Python 3.6‑3.10, GPU/TPU optional	Reliable for real‑world services

---

Typical Use Cases

• Text preprocessing pipelines for downstream ML (e.g., tokenization + POS + NER before classification).
• Information extraction such as named‑entity recognition, relation extraction, and semantic role labeling in news, finance, or legal documents.
• Multilingual content analysis – sentiment, language detection, or key‑phrase extraction across dozens of languages.
• Academic research – fast baseline models for parsing, AMR, or coreference without building models from scratch.
• Production services – expose the RESTful endpoint to mobile apps or micro‑services that need low‑latency NLP without managing GPUs.

---

Getting Started

1. Install

# Native Python API (full models, GPU optional)
pip install hanlp

# Lightweight RESTful client (any language)
pip install hanlp_restful   # Python example

2. Quick Python example (native API)

import hanlp
# Load a multilingual joint model (small Chinese ELECTRA backbone)
HanLP = hanlp.load(
    hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH)

sentences = [
    "2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。",
    "阿婆主来到北京立方庭参观自然语义科技公司。"
]
result = HanLP(sentences)
print(result)          # JSON‑compatible output
HanLP.pretty_print()   # visual console view

3. Quick Python example (RESTful API)

from hanlp_restful import HanLPClient
client = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh')
print(client.parse("HanLP提供了强大的中文分词和命名实体识别功能。"))

4. Other languages

The same client libraries exist for Go and Java; just add the dependency and create a HanLPClient with the server URL and optional API key.

---

Documentation & Community

• Docs & demos – https://hanlp.hankcs.com/docs/ (including Jupyter notebooks for each task)
• Model catalogue – https://hanlp.hankcs.com/docs/api/hanlp/pretrained/
• Forum & papers – https://bbs.hankcs.com/
• Citation – If you use HanLP in research, cite the EMNLP 2021 paper linked in the README.

---

TL;DR

HanLP is a mature, multilingual NLP library that offers both a tiny RESTful service and a full‑featured native Python API. It ships with dozens of pretrained models for everything from tokenization to semantic parsing, supports custom dictionaries, and is designed for both research prototyping and production deployment.