Hugging Face TextImage Augmentation pipeline release

TL;DR

Hugging Face and Albumentations AI released a TextImage Augmentation pipeline that simultaneously modifies document images and their embedded text, enabling realistic synthetic data generation and robust fine‑tuning of vision‑language models on limited document datasets.

Motivation: Augmentation Must Preserve Text

Document images contain dense textual information, so traditional image‑only augmentations (e.g., resizing, blurring, background changes) often degrade OCR performance. The blog post argues that effective data augmentation for such data must preserve the integrity of the text while diversifying visual appearance. This need becomes critical when fine‑tuning vision‑language models (VLMs) on small document corpora, where missing or corrupted text can hinder learning.

Introduction: A Multimodal Augmentation Pipeline

The new pipeline, co‑developed with Albumentations AI, treats document augmentation as a multimodal problem: it applies transformations to both the image pixels and the associated text annotations in lockstep. The approach builds on a prior Hugging Face blog post that hypothesized joint text‑image augmentations improve VLM pre‑training. Detailed parameter specifications and example use cases are documented on the Albumentations AI website.

Method: From Bounding Boxes to In‑painted Text

  1. Line Selection – Randomly choose document lines based on a fraction_range hyperparameter that controls the proportion of bounding boxes to modify.
  2. Text Augmentation – Apply one of several NLP‑style operations to the selected lines:
    • Random Insertion (stop‑word insertion)
    • Random Deletion
    • Random Swap
    • Stopword Replacement
  3. Image Update – Black out the original text region, then inpaint it with the newly generated text. Font size is derived from the bounding box height via font_size_fraction_range. The pipeline returns both the altered image and the updated text metadata, enabling downstream training pipelines to consume the transformed pairs.

Main Features of TextImage Augmentation

1. Synthetic Text Overlay

Render arbitrary text on any background image, creating fully synthetic document samples. This mirrors techniques such as SynthDOG from the OCR‑free document understanding transformer.

2. Augmented Text Overlay

Apply text‑level perturbations while preserving visual realism. Supported operations include:

  • Random Deletion – removes words at random.
  • Random Swapping – swaps word order within a line.
  • Stop‑word Insertion – injects common stop words (e.g., "the", "and").

These augmentations can be combined with any Albumentations image transforms (e.g., color jitter, affine warps), and the transformed text can be extracted via the overlay_data field.

Note: The earlier version of the repo included synonym replacement, but it was removed due to significant runtime overhead.

Installation

pip install -U pillow
pip install albumentations
pip install nltk

import albumentations as A, cv2, json, nltk
from matplotlib import pyplot as plt
nltk.download('stopwords')
from nltk.corpus import stopwords

Visualization Helper

def visualize(image):
    plt.figure(figsize=(20,15))
    plt.axis('off')
    plt.imshow(image)

Loading Document Data

The pipeline expects line‑level bounding boxes (normalized Pascal VOC format) and corresponding text. Example datasets:

  • pixparse/idl-wds
  • pixparse/pdfa-eng-wds
bgr_image = cv2.imread('examples/original/fkhy0236.tif')
image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
with open('examples/original/fkhy0236.json') as f:
    labels = json.load(f)
font_path = '/usr/share/fonts/truetype/liberation/LiberationSerif-Regular.ttf'
visualize(image)

Metadata preparation converts normalized boxes to absolute coordinates:

def prepare_metadata(page, h, w):
    meta = []
    for txt, box in zip(page['text'], page['bbox']):
        left, top, w_norm, h_norm = box
        meta.append({
            'bbox': [left, top, left + w_norm, top + h_norm],
            'text': txt
        })
    return meta

page = labels['pages'][0]
metadata = prepare_metadata(page, *image.shape[:2])

Text Augmentation Examples

Random Swap

transform = A.Compose([
    A.TextImage(font_path=font_path, p=1, augmentations=['swap'],
                 clear_bg=True, font_color='red',
                 fraction_range=(0.5,0.8),
                 font_size_fraction_range=(0.8,0.9))
])
out = transform(image=image, textimage_metadata=metadata)
visualize(out['image'])

Random Deletion

transform = A.Compose([
    A.TextImage(font_path=font_path, p=1, augmentations=['deletion'],
                 clear_bg=True, font_color='red',
                 fraction_range=(0.5,0.8),
                 font_size_fraction_range=(0.8,0.9))
])
out = transform(image=image, textimage_metadata=metadata)
visualize(out['image'])

Random Insertion (Stop‑word Insertion)

stops = stopwords.words('english')
transform = A.Compose([
    A.TextImage(font_path=font_path, p=1, augmentations=['insertion'],
                 stopwords=stops, clear_bg=True, font_color='red',
                 fraction_range=(0.5,0.8),
                 font_size_fraction_range=(0.8,0.9))
])
out = transform(image=image, textimage_metadata=metadata)
visualize(out['image'])

Combining with Other Albumentations Transforms

A complex pipeline can interleave text insertion with image‑level augmentations such as PlanckianJitter (color balance) and Affine (scaling/rotation):

transform_complex = A.Compose([
    A.TextImage(font_path=font_path, p=1, augmentations=['insertion'],
                 stopwords=stops, clear_bg=True, font_color='red',
                 fraction_range=(0.5,0.8),
                 font_size_fraction_range=(0.8,0.9)),
    A.PlanckianJitter(p=1),
    A.Affine(p=1)
])
out = transform_complex(image=image, textimage_metadata=metadata)
visualize(out['image'])

Extracting the Altered Text

The overlay_data field contains a list of dictionaries with:

  • bbox_coords: pixel coordinates of the modified region
  • text: the new augmented text
  • original_text: the source line
  • bbox_index: index in the original metadata list
  • font_color: rendering color
print(out['overlay_data'])

Sample output shows swapped or inserted words while preserving layout.

Synthetic Data Generation

Beyond perturbing existing documents, the pipeline can render arbitrary text on any template:

template = cv2.imread('template.png')
image_template = cv2.cvtColor(template, cv2.COLOR_BGR2RGB)
transform = A.Compose([
    A.TextImage(font_path=font_path, p=1, clear_bg=True,
                 font_color='red', font_size_fraction_range=(0.5,0.7))
])
metadata = [
    {'bbox':[0.1,0.4,0.5,0.48], 'text':'Some smart text goes here.'},
    {'bbox':[0.1,0.5,0.5,0.58], 'text':'Hope you find it helpful.'}
]
out = transform(image=image_template, textimage_metadata=metadata)
visualize(out['image'])

This capability enables large‑scale creation of labeled document images without manual annotation.

Conclusion

The TextImage Augmentation library provides a unified, multimodal augmentation workflow for document images. By coupling classic NLP perturbations (random insertion, deletion, swap, stop‑word replacement) with Albumentations’ powerful image transforms, practitioners can generate diverse, realistic training data and improve VLM fine‑tuning on scarce document corpora. Detailed parameter documentation and examples are available on the Albumentations AI site.

References

  • Kim, G., Hong, T., Yim, M., et al. OCR‑free Document Understanding Transformer, ECCV 2022.

Installation Summary

pip install -U pillow albumentations nltk

Sources