kesai-labs/py123d

py123d – Unified Autonomous‑Driving Dataset Library

What it is – py123d is a Python package that lets you work with many of the major autonomous‑driving datasets (Argoverse 2, nuScenes, Waymo, nuPlan, KITTI‑360, PandaSet, etc.) through a single, consistent API. It downloads raw data, converts it into a columnar Apache Arrow format, and then provides zero‑copy, memory‑mapped access to cameras, lidars, maps, and annotations.

Why it matters – Autonomous‑driving research often has to juggle different file layouts, sensor codecs, and label schemas. py123d removes that friction, enabling you to:

• Pull datasets directly from their official sources via a CLI.
• Store everything in a compact Arrow file that can be read instantly without loading the whole dataset into RAM.
• Switch between datasets without changing your training or evaluation code.
• Visualise scenes instantly with the built‑in Viser 3‑D viewer or simple Matplotlib plots.

Key features

Feature	Details
Dataset download & conversion	One‑line CLI (py123d-conversion …) fetches raw logs, converts them to Arrow, and cleans up the originals.
Hydra‑based pipelines	YAML configs let you describe complex conversion steps and reuse them across projects.
Zero‑copy Arrow storage	Columnar, memory‑mapped reads give fast, low‑memory access to billions of points or frames.
Multiple sensor codecs	Supports MP4/JPEG/PNG for cameras and LAZ/Draco/Arrow IPC for lidar point clouds.
No sensor duplication	Converted logs keep references to the original files, avoiding double storage.
Unified API	Same Python calls (scene.cameras(), scene.lidar(), scene.map(), scene.labels()) work for any supported dataset.
Built‑in visualisation	Interactive 3‑D viewer (Viser) and Matplotlib helpers for quick inspection.
Extensible modalities	Recent releases add depth cameras, radar, per‑pixel segmentation, route information, and pose‑uncertainty fields.
Multi‑dataset training support	Companion repo py123d_garage offers training loops, NAVSIM metrics, and simulation bridges to CARLA/AlpaSim.

Installation

# Core library
pip install py123d

# Extras for a specific dataset (install only what you need)
# e.g. Argoverse 2 support
pip install py123d[av2]

The package follows standard Python versioning and is hosted on PyPI.

Quick demo (Argoverse 2)

# 1️⃣ Install the AV2 extra
pip install py123d[av2]

# 2️⃣ Choose where converted data will live
export PY123D_DATA_ROOT=/my/py123d_data

# 3️⃣ Download 3 validation logs and convert them to Arrow
py123d-conversion dataset=av2-sensor-stream \
  dataset.parser.splits='[av2-sensor_val]' \
  dataset.parser.downloader.num_logs=3

# 4️⃣ Launch the interactive viewer
py123d-viser scene_filter=av2-sensor

Open http://localhost:8080 in a browser to explore the scenes.

Supported datasets – The library currently handles a wide range of manual, auto‑labeled, and synthetic collections, including:

• Manual: nuScenes, Waymo Open Perception, Argoverse 2 Sensor, PandaSet, KITTI‑360
• Auto‑labeled: Waymo Open Motion, nuPlan (and mini), NVIDIA Physical AI AV, NCore
• Synthetic: CARLA, L3AD (CARLA‑derived) Each entry lists available sensors (cameras, lidar, radar) and annotation types (3‑D boxes, traffic lights, maps, segmentation, etc.).

Documentation & community – Full API reference, conversion guides, and contribution instructions are available at https://kesai.eu/py123d/. The project is open‑source under the Apache‑2.0 license and welcomes extensions for new datasets or modalities.

---

Citation If you use py123d in research, please cite the accompanying paper:

@article{Dauner2026ARXIV,
  title={123D: Unifying Multi-Modal Autonomous Driving Data at Scale},
  author={Dauner, Daniel and Charraut, Valentin and Berle, Bastian and Li, Tianyu and Nguyen, Long and Wang, Jiabao and Jing, Changhui and Igl, Maximilian and Caesar, Holger and Ivanovic, Boris and Geiger, Andreas and Chitta, Kashyap},
  journal={arXiv preprint arXiv:2605.08084},
  year={2026}
}