jdkato/prose

What it solves

It provides a comprehensive set of natural language processing (NLP) tools for the Go programming language, allowing developers to analyze English text without relying on external services or C-based dependencies (cgo).

How it works

The library implements a pipeline of NLP tasks. It uses the Punkt algorithm for sentence segmentation and the Penn Treebank tag set for part-of-speech tagging. It includes pre-trained models for tagging and named entity recognition (NER) that are loaded into memory once per process. To keep binary sizes small, these models are separated into their own packages, allowing the Go linker to exclude unused components.

Who it’s for

Go developers who need to perform basic NLP tasks like tokenization, entity extraction, and readability scoring in a standalone, pure-Go environment.

Highlights

• Pure Go: No cgo or external API calls required.
• Byte-accurate offsets: Every token, sentence, and entity maintains a precise byte offset to the original source text.
• Modular Design: Components like the tagger and entity recognizer can be imported independently to reduce binary overhead.
• Readability Metrics: Built-in support for Flesch–Kincaid, Gunning fog, SMOG, and Coleman–Liau scores.
• Concurrent Safety: All core analysis tools are safe for concurrent use.