tensorflow/data-validation

Library for exploring and validating machine learning data

TensorFlow Data Validation (TFDV)

What it is – A Python library that helps you explore, understand, and validate tabular data used for training TensorFlow models. It plugs directly into TensorFlow Extended (TFX) pipelines but can also be used as a stand‑alone tool.

Core capabilities

  • Scalable statistics – Computes feature‑wise summary stats (mean, min/max, histograms, etc.) on large datasets using Apache Beam.
  • Schema inference & enforcement – Automatically generates a TensorFlow Metadata schema describing expected types, value ranges, vocabularies, and required fields, then checks incoming data against it.
  • Anomaly detection – Flags missing features, out‑of‑range values, unexpected types, and other data quality issues.
  • Visualization – Integrates with the Facets UI and provides built‑in viewers for statistics, schemas, and detected anomalies.
  • Distributed execution – Runs locally for small jobs or on Cloud Dataflow / other Beam runners for big data.

Typical workflow

  1. Ingest data (CSV, TFRecord, Parquet, etc.) into a Beam pipeline.
  2. Generate statistics with tfdv.generate_statistics_from_csv (or similar).
  3. Infer a schema from the stats using tfdv.infer_schema.
  4. Validate new data against the schema via tfdv.validate_statistics.
  5. Inspect results in the Facets UI or programmatically handle anomalies.

Installation

pip install tensorflow-data-validation   # stable release
# or, for the latest nightly builds:
export TFX_DEPENDENCY_SELECTOR=NIGHTY
pip install --extra-index-url https://pypi-nightly.tensorflow.org/simple tensorflow-data-validation

Docker images are provided for reproducible builds on Linux.

Dependencies – Requires TensorFlow, Apache Beam (for distributed processing), and Apache Arrow (for efficient in‑memory columnar representation).

Who should use it – Data engineers and ML practitioners who need reliable, automated data quality checks as part of model training or production monitoring, especially when working with large‑scale TensorFlow pipelines.

Further reading – Get‑started guide, example Colab notebook, API docs, and the SysML’19 paper describing the underlying techniques are linked in the README.

Related

  • Project
    tensorflow/model-analysis

    TensorFlow Model Analysis (TFMA) is a library for large‑scale, slice‑aware evaluation of TensorFlow models. It runs distributed Beam pipelines, visualizes results in Jupyter, and integrates with TFX/Kubeflow pipelines.

  • Project
    LongxingTan/Time-series-prediction

    A TensorFlow-based time series package that provides a unified interface for prediction, classification, and anomaly detection using state-of-the-art deep learning models.

  • Project
    tensorflow/transform

    A library for preprocessing data with TensorFlow that supports full-pass operations and exports transformations as a graph to prevent training-serving skew.

  • Project
    tensorflow/java

    TensorFlow‑Java provides Maven‑publishable Java/JVM bindings for TensorFlow, including low‑level JNI wrappers (`tensorflow‑core`), a high‑level neural‑network API (`tensorflow‑framework`), and an independent ndarray library. It supports Linux (x86_64, arm64) and macOS (Apple Silicon) binaries, with optional GPU builds, and integrates with standard Java build tools.

  • Dispatch
    Deploying TensorFlow Vision Models in Hugging Face with TF Serving

    Hugging Face provides a guide on deploying TensorFlow vision models, such as the Vision Transformer (ViT), using TensorFlow Serving to expose them as REST or gRPC endpoints.