kermitt2/delft

What it solves

DeLFT is a deep learning framework designed for text processing, specifically targeting sequence labeling (like named entity tagging) and text classification. It addresses the limitation of many NLP tools that only handle simple text, by providing native support for "rich text"—text associated with layout information, font styles, and positions within structured documents like PDFs or HTML files.

How it works

Built on Keras and TensorFlow, the framework implements various state-of-the-art deep learning architectures, including RNNs and Transformers. It integrates HuggingFace transformers as Keras layers and uses dynamic data generators to handle large datasets that cannot fit in memory. To optimize for production, it reduces the size of RNN models by removing word embeddings, significantly shrinking model footprints (often to under 2 MB).

Who it’s for

It is intended for developers and researchers who need to perform high-performance sequence labeling or text classification on complex, structured documents, and those who want a reproducible benchmarking environment for NLP models.

Highlights

• Rich Text Support: Handles tokens with layout and symbolic contextual information from structured documents.
• Model Compression: Significantly reduces RNN model sizes (e.g., from 230 MB to 1.8 MB) for easier deployment.
• Production Ready: Optimized for performance and robustness with efficient loading of pre-trained embeddings via LMDB.
• Comprehensive Evaluation: Includes a built-in framework for standard metrics and n-fold cross-validation.
• HuggingFace Integration: Seamlessly integrates transformer-based models as Keras layers.