dsta022/Loop-Engineering-for-VLA

Loop Engineering for VLAnything – A Toolkit for Building High‑Quality Vision‑Language‑Action Datasets

What it is

• An open‑source Python toolkit that lets researchers record, merge, audit, enrich, and iterate on multimodal robot‑learning datasets built for vision‑language‑action (VLA) models such as those in the lerobot ecosystem.
• It works with standard RGB‑only LeRobot datasets and RGB‑D datasets that include loss‑less depth side‑cars, plus a work‑in‑progress visuo‑tactile mode.
• The whole pipeline is model‑free by default – you can plug in evaluators, annotators, or policy trainers later without pulling in heavy model weights.

Why it matters

• High‑quality VLA data is hard to obtain; small errors in video, depth, or action logs can cripple downstream training.
• The toolkit treats every artifact (videos, joint states, language captions, human feedback) as a first‑class side‑car that never mutates the original recordings, making the data audit‑friendly and reproducible.
• It provides a conservative, rule‑based quality audit that automatically drops clearly broken episodes while flagging borderline cases for human review.

Core components

Area	Entry‑point (CLI)	What it does
Recording	vla-merge, vla-audit, … (see table below)	Framework‑free recorder that talks directly to SO‑100/SO‑101 robot arms and to Orbbec, Intel RealSense, or generic OpenCV cameras. Writes LeRobot‑v3.0‑compatible datasets with optional depth side‑cars and human‑guided policy feedback.
Merging	vla-merge	Combines many LeRobot datasets (RGB‑only, RGB‑D, etc.) into a single physical dataset, rewriting indices, metadata, and depth files.
Auditing	vla-audit	Deterministic structural checks (metadata, timestamps, video decode, depth PNG integrity) plus optional semantic checks that run a policy checkpoint or custom evaluator to score how well the episode follows its task description.
Cleaning	vla-clean	Builds a clean dataset by applying the keep/review/drop lists produced by the audit.
Language enrichment	vla-enrich	Adds trajectory‑language side‑cars (subtasks, events, state descriptions) using a built‑in kinematic annotator or any user‑provided language annotator plugin.
Feedback collection	vla-feedback-build	Captures human interventions on a deployed policy, stores them as side‑cars, and later merges them into a training set.
Policy iteration	vla-iterate	Simple train‑evaluate‑promote‑deploy loop that can be hooked to any lerobot checkpoint.
Upload	vla-push	Pushes the final dataset to Hugging Face (the repo ships a ready‑to‑use HF dataset: DerekLX/lerobot_derek_depth).
Utility tools	vla-completeness, vla-calibrate, vla-depth-vis, vla-task-studio, vla-language-studio	Helpers for dataset sanity‑checks, threshold calibration, depth visualisation, and interactive profile/annotation editing.

How to get started

1. Install Python 3.12+, create a conda env, and add ffmpeg (required for video handling).
2. Install the repo in editable mode with the recording extras you need, e.g.

   conda create -n vla python=3.12 -y && conda activate vla
   conda install -c conda-forge ffmpeg -y
   pip install -e "[record]"   # adds camera‑SDK optional deps later with [realsense] or [orbbec]
   

3. Use the record/ scripts to capture new episodes (RGB or RGB‑D). Example for an Orbbec camera:

   python record/rgbd_record/find_cameras.py orbbec
   python record/rgbd_record/record.py --config_path=record/rgbd_record/configs/record_rgbd.yaml
   

4. Merge multiple raw datasets into one unified folder with vla-merge.
5. Run a structural audit (vla-audit) to generate keep_episodes.txt, review_episodes.txt, and drop_episodes.txt.
6. (Optional) Plug a semantic evaluator – e.g. the built‑in semantic_backends.vla_checkpoint_evaluator – to score episodes against a policy checkpoint.
7. Enrich the kept episodes with language side‑cars via vla-enrich.
8. If you have collected human feedback on a deployed policy, build the feedback dataset with vla-feedback-build and feed it back into vla-iterate for the next training round.
9. Push the final, audited dataset to Hugging Face with vla-push.

Who should use it

• Researchers building vision‑language‑action or embodied AI models who need a reproducible data‑engineering pipeline.
• Robotics labs that already use the lerobot data format and want to add depth or tactile side‑cars without rewriting their existing code.
• Teams that want a closed‑loop data‑collection loop: record → audit → enrich → train → collect feedback → repeat.

Key design principles

1. Conservative decisions – only hard failures are dropped; ambiguous cases are marked for review.
2. Side‑car‑first – depth, language, feedback, and even training weights live outside the core dataset, preserving the original recordings.
3. Pluggable, model‑free by default – the base install ships no weights; any evaluator, annotator, or policy can be added via a simple module:attribute plugin string.

Documentation & community

• A full static website (https://dsta022.github.io/Loop-Engineering-for-VLA/) hosts the playbooks, plugin contracts, and audit strategy docs.
• CI runs unit tests on synthetic data for both Linux and Windows.
• The repository includes example configuration files, a synthetic test suite, and a ready‑to‑use Hugging Face dataset.

---

All details above are taken directly from the repository’s README; no additional features have been inferred.