cltk/cltk

What it solves

CLTK provides natural language processing (NLP) tools specifically designed for pre-modern and historical languages, which often have different linguistic characteristics and researcher goals than living languages. It fills a gap where most standard NLP software is built for modern languages.

How it works

The toolkit uses a modular processing pipeline that allows users to choose between different backends for language annotation. It supports multiple backends, including:

• GenAI/LLMs: Support for OpenAI (cloud-based) and Ollama (local LLMs like Llama 3.1 or Qwen 2.5).
• Discriminative NLP: Support for Stanford Stanza.

Users can specify the language (e.g., lati1261 for Latin) and the choose the backend to perform NLP tasks.

Who it’s for

Researchers and scholars of pre-modern languages who need a standardized, Python-based framework for analyzing historical texts.

Highlights

• Multi-backend support: Integrates with LLMs (OpenAI, Ollama) and traditional NLP tools (Stanza).
• Modular pipeline: Balances algorithmic diversity with pre-configured defaults.
• Broad language support: Provides pipelines and models for nearly 20 languages.
• Local LLM integration: Allows the use of local models via Ollama to ensure data privacy or offline access.