Microduck RL – Sim‑to‑real training for a tiny bipedal robot

What it is – A collection of MuJoCo‑Warp environments and training scripts that teach a 800 g, 25 cm tall biped (the Microduck) to walk, stand, roll, skate, kick, and more. Policies are learned with PPO, exported to ONNX, and then run on the physical robot by the companion microduck runtime.

---

Why it matters

• Full sim‑to‑real pipeline – uses the BAM actuator model (voltage‑controlled Dynamixel XL330) with realistic friction, backlash, and domain randomisation. This fidelity is the biggest source of the sim‑real gap for such a small, servo‑driven robot.
• Hot‑swapable policies – All policies share a 61‑dimensional observation vector, so the robot can switch between walking, recovery, tricks, etc., without restarting the daemon.
• Open‑source reproducibility – Everything from the MJCF robot files (generated from Onshape) to the training commands, export scripts, and publishing workflow is provided.

---

How it works (high‑level flow)

1. Environment definition – src/mjlab_microduck/tasks/*_env_cfg.py describe each task (e.g., Mjlab-Velocity-Flat-MicroDuck). They set up terrain, reward functions, and randomisation flags.
2. Training – Run uv run train <task> --env.scene.num-envs 4096. PPO runs on the GPU via MuJoCo Warp; a typical gait finishes in 1‑2 h.
3. Inspection – uv run play … visualises a checkpoint in the MuJoCo viewer; you can also record videos.
4. Export – scripts/export.py converts the checkpoint to an ONNX graph that already contains the observation normaliser.
5. Publish – uv run publish … uploads the ONNX file plus a manifest to the Hugging Face Hub, making it installable on any Microduck robot.
6. Deployment – On the robot, robotctl policy add … and robotctl policy load walk … load the policy; the daemon swaps it in at runtime.

---

Key components

Component	Role
BAM actuator model	Simulates voltage control, back‑EMF, Coulomb & Stribeck friction, and optional ±1° gear play (backlash).
MJCF robot files	Exported from Onshape; several variants (robot_walk.xml, robot_groundcontact.xml, robot_groundcontact_rollers.xml, plus backlash versions).
Task registry (src/mjlab_microduck/tasks/__init__.py)	Registers every environment, including automatically generated backlash twins.
Training CLI (train_cli.py)	Thin wrapper around the mjlab training framework; identical to upstream mjlab CLI.
Export / inference scripts	scripts/export.py produces ONNX; scripts/infer_policy.py runs a policy in CPU MuJoCo for quick testing.
Publish tool	Packages ONNX + manifest.json + README and pushes to Hugging Face, handling validation (shape, NaNs, normaliser).
Tests	Small pytest suite checks reward sign conventions, joint‑index consistency, and NaN guards.

---

Getting started (quick steps)

# clone & enter
git clone https://github.com/pollen-robotics/microduck_rl && cd microduck_rl

# train a walking gait (GPU required)
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096

# watch it
uv run play Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <entity/project/run_id>

# export to ONNX
uv run scripts/export.py Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <...>

# optional: publish to Hugging Face
uv run publish --onnx output.onnx --repo <user>/microduck-walk --kind perpetual --slot walk

No GPU? Add --hf-jobs to off‑load training to Hugging Face Jobs.

---

Typical use‑cases

• Research – Experiment with domain randomisation, actuator modelling, or new reward designs on a real‑world biped.
• Education / demos – Show a small robot performing tricks (roulade, roller‑skating) with a few lines of code.
• Community sharing – Publish custom gaits or tricks to the Hugging Face Hub; anyone with a Microduck can install them instantly.

---

License & attribution

• Code: Apache 2.0.
• 3‑D meshes: CC‑BY‑NC‑SA 4.0 (see LICENSE-MODELS.md).

---

Microduck RL is a concrete, end‑to‑end example of modern sim‑to‑real reinforcement learning for a physical robot, complete with realistic actuator physics, domain randomisation, and a streamlined publishing workflow.