rerun-io/rerun

Visualize, query, and stream to train on multimodal robotics data.

What it solves

Rerun is a visual and temporal debugger for multimodal data. It solves the problem of understanding and improving complex processes—such as those in robotics, computer vision, and simulation—where traditional text logs or standard debuggers cannot explain why a system is failing (e.g., why a robot thinks a wall is a door).

How it works

It functions as a data layer that ingests multi-rate, multimodal data (including images, point clouds, tensors, and joint states) from various sources like robot logs, simulations, and web video. This data is stored in a shared columnar storage system built in Rust, which allows the data to be queried via dataframes or SQL and streamed directly into training pipelines. A built-in viewer renders these streams in sync and in real-time, allowing users to scrub through episodes and compare sensors side-by-side.

Who it’s for

It is designed for developers and researchers working with sensors, 2D/3D state evolving over time, and physical AI, specifically those in robotics, simulation, and computer vision.

Highlights

  • Multimodal Support: Handles images, point clouds, time series, tensors, transforms, and joint states.
  • Time-Aware Database: Includes a viewer that allows users to go back in time to pinpoint exactly when and when and why a failure occurred.
  • Direct Training Integration: Streams dataset mixes directly to training, eliminating the need for export jobs or stale copies.
  • Cross-Language SDKs: Provides SDKs in Python, Rust, and C++.
  • Queryable Storage: Data is queryable using dataframes or SQL.

Related

  • Project
    Flaminis/Dalaran

    A robotics-first observability and visualization stack for multimodal time-series data, providing synchronized 3D/2D viewing and Arrow-backed storage for robot sensor streams.

  • Project
    Eventual-Inc/Daft

    A high-performance, Rust-powered data engine designed to process multimodal AI workloads, allowing users to handle images, audio, and video alongside structured data at scale.

  • Project
    jolibrain/deepdetect

    A deep learning runtime and REST server that provides a unified API for training and inference across images, text, and tabular data.

  • Project
    Alibaba-NLP/VRAG

    A multi-turn multimodal RAG framework that uses agentic reinforcement learning to enable VLMs to reason and retrieve information across text, images, and videos.

  • Project
    duoan/mega-data-factory

    A distributed, high-throughput pipeline for processing web-scale multimodal datasets, featuring Rust-accelerated operators for filtering, scoring, and deduplication.