google/seqio

What it solves

SeqIO is a library designed to simplify the processing of sequential data for sequence-to-sequence models. It provides a unified way to handle task-based datasets, preprocessing pipelines, and evaluation metrics, reducing the boilerplate required to prepare data for models like T5 and other sequence models across different frameworks.

How it works

SeqIO uses tf.data.Dataset to create scalable data pipelines, but it is designed to be framework-agnostic, allowing datasets to be converted to numpy iterators for use with JAX or PyTorch. The core abstraction is the Task, which bundles together a raw data source, sequential preprocessing steps, a vocabulary for tokenization, a postprocessor for evaluation, and specific metrics.

Users can define multiple Task objects and combine them into a Mixture to sample examples from different tasks at specified rates. The pipeline follows a flow of: defining a task/mixture $\rightarrow$ applying preprocessors $\rightarrow$ tokenizing via features $\rightarrow$ generating a final dataset.

Who it’s for

It is intended for researchers and developers working with sequence models (text, audio, or sequence-represented images) who need a scalable, flexible way to manage multi-task datasets and evaluation pipelines.

Highlights

• Framework Agnostic: Built on TensorFlow's data API but compatible with JAX and PyTorch via numpy iterators.
• Task Abstraction: Combines data loading, preprocessing, tokenization, and evaluation into a single object.
• Flexible Mixing: Supports Mixture registries to combine multiple tasks with customizable sampling rates.
• Scalable Pipelines: Leverages tf.data.Dataset for efficient, parallelized data loading and preprocessing.