tensorflow/model-analysis

TensorFlow Model Analysis (TFMA)

What it is – TFMA is a Python library that lets you evaluate TensorFlow models on large datasets. It re‑uses the same metric definitions you wrote for training, runs the evaluation in a distributed fashion (via Apache Beam), and lets you slice results (e.g., by feature values) and explore them in Jupyter notebooks.

Key capabilities

• Scalable evaluation – works on local machines or on distributed runners such as Google Cloud Dataflow.
• Slice‑based metrics – compute metrics for arbitrary data slices (e.g., per‑country, per‑age‑group) without extra code.
• Visualization – built‑in Jupyter widgets and a slicing‑metrics browser to inspect results interactively.
• Integration points – works with TFX pipelines, Kubeflow Pipelines, and can export a self‑contained HTML report.

Installation

# standard install from PyPI
pip install tensorflow-model-analysis

# nightly builds (if you need the very latest)
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-model-analysis

# install directly from the repo (e.g., a specific tag)
pip install git+https://github.com/tensorflow/model-analysis.git@v0.21.3#egg=tensorflow_model_analysis

TFMA does not pull TensorFlow automatically; you must have a compatible TensorFlow version installed.

Typical workflow

1. Define metrics in your training code (e.g., tf.keras.metrics.AUC).
2. Create an EvalConfig that points to those metrics and optionally defines slicing specs.
3. Run TFMA – either via the Python API (tfma.run_model_analysis) or as a TFX component. Under the hood it builds an Apache Beam pipeline that reads your examples, applies the model, and aggregates the metrics.
4. Explore results – open the generated tfma notebook or use the tfma widget in JupyterLab to browse slice‑wise tables and charts.

Jupyter integration

• Install the matching JupyterLab extension (e.g., jupyter labextension install tensorflow_model_analysis@0.32.0).
• Enable the notebook extensions for classic Jupyter (jupyter nbextension enable --py tensorflow_model_analysis).
• Use the provided tfma widgets to render interactive tables directly in notebooks.

Dependencies

• TensorFlow – the core ML framework.
• Apache Beam – powers the distributed computation (runs locally by default, can target Dataflow, Spark, Flink, etc.).
• Apache Arrow – used for efficient in‑memory columnar data handling.

Getting started – see the repository’s get‑started guide: https://github.com/tensorflow/model-analysis/blob/master/g3doc/get_started.md.

Version compatibility – the README includes a detailed matrix linking TFMA releases to specific versions of TensorFlow, Apache Beam, PyArrow, and other TFX components. Choose a TFMA version that matches the rest of your stack.

Support – questions are answered on Stack Overflow under the tensorflow-model-analysis tag.