EmilStenstrom/conllu

What it solves

CoNLL-U Parser is a lightweight Python library designed to handle CoNLL-U formatted strings, which are commonly used as output for natural language processing (NLP) tasks. It converts these formatted strings into structured Python data types (dictionaries and lists) that are easier to programmatically manipulate.

How it works

The library provides two primary parsing methods:

• parse(): Converts CoNLL-U data into a flat list of sentences, where each sentence is a TokenList containing tokens represented as ordered dictionaries.
• parse_tree(): Converts the data into a nested TokenTree structure, which represents the dependency tree of a sentence based on the head column of the CoNLL-U file.

For large files, the library offers incremental parsing methods (parse_incr and parse_tree_incr) to avoid loading the entire dataset into memory.

Who it’s for

Developers and researchers working with NLP datasets in the CoNLL-U format, specifically those who need to parse, filter, modify, and serialize data back into the CoNLL-U format.

Highlights

• Zero Dependencies: The library is written in ~300 lines of code and has no external dependencies.
• Flexible Parsing: Supports custom field names, custom field parsers, and custom metadata parsers to handle non-standard variations of the CoNLL-U format.
• TokenList Utilities: Includes built-in filtering capabilities (including lambda-based filtering) and the ability to serialize data back to text.
• Dependency Tree Support: Can transform flat token lists into hierarchical dependency trees for easier traversal.
• Strong Testing: Features 100% test branch coverage and mutation testing.