X-Embodiment Language-Grounded Manipulation Benchmark
A ManiSkill3 benchmark foundation for measuring language-grounded manipulation transfer between a tabletop arm and a humanoid upper body through a shared canonical action interface.
Role — Creator — benchmark architecture, environments, evaluation design, reproducibility engineering
- Robot policies are increasingly expected to generalize across tasks, scenes, and bodies. This benchmark asks a narrower, measurable question: when language, task structure, and policy architecture are held fixed, how much manipulation performance survives a change in embodiment — and where exactly does it drop?
- Simulation benchmark · cross-embodiment evaluation infrastructure
- Cross-embodiment transfer is a central open question for physical AI. Claims about it need benchmarks with leakage-proof splits and honest failure accounting — the infrastructure this project builds before drawing any conclusions.
- Solo project, developed independently alongside the Booster K1 research.
Overview
An in-progress benchmark that treats embodiment transfer as a measurable systems problem. A shared language-conditioned policy stack emits actions in a canonical 21-dimensional interface; thin robot-specific adapters translate them for either a Panda/Franka-style tabletop arm or a Unitree G1 humanoid upper body in ManiSkill3. Identical tasks, language splits, and evaluation protocol on both bodies make the interesting quantity directly computable: how much performance survives when only the robot changes. Phase A — environments, demonstration pipeline, baseline scaffolding, and the reproducibility layer — is complete and frozen at tag v0.3.1; Phase B, real training and evaluation on an RTX 5090 Linux workstation, is pending, and with it all empirical results.
A language instruction is drawn from a train or held-out paraphrase split and fed to the shared policy stack, which emits actions in the canonical 21-D interface. A robot adapter translates them for the Panda or G1 environment, and evaluation episodes stream into transfer, failure, and artifact reports. The same path runs unchanged for either body — that symmetry is the experiment.
- Language instruction — paraphrase / holdout splits
- Shared policy stack — BC baseline · PPO scaffold
- Canonical 21-D action interface
- Robot adapter — Panda | Unitree G1
- Eval episodes → transfer, failure & artifact reports
Contributions
- Designed the cross-embodiment benchmark structure around a shared canonical 21-dimensional action interface, with thin per-robot adapters for Panda, Panda-stick, and the Unitree G1 upper body.
- Implemented language-conditioned task pathways for both embodiments in ManiSkill3, with leakage-proof held-out paraphrase and held-out color/object splits designed in before any training.
- Built the HDF5 demonstration pipeline with validation, plus a Behavior Cloning baseline and PPO fine-tuning scaffolding.
- Added episode-level evaluation records, a failure taxonomy, transfer-drop and retention metrics, and artifact-pack generation — rehearsed end-to-end on deterministic synthetic data.
- Hardened the repo for reproducibility: 110 fast + 5 simulation tests, full smoke checks, documentation, and a Linux Phase B runbook; frozen pre-Linux at tag v0.3.1.
Evidence & evaluation
Evidence
Test suite
attached110 fast + 5 simulation tests passing; full smoke check green.
Reproducibility freeze
attachedTagged v0.3.1 pre-Linux — pinned pipeline, docs, and a Phase B runbook.
Artifact-pack rehearsal
attachedReporting pipeline rehearsed end-to-end on deterministic synthetic toy data — explicitly not real results.
Benchmark results
pendingAll real training and evaluation numbers await Phase B on the RTX 5090 Linux workstation.
Rollout videos
pendingDemo videos will accompany Phase B evaluation runs.
Metrics
Not yet measured
Pending Phase B — no real numbers exist yet.
Not yet measured
Arm-to-humanoid retention ratio; pending Phase B.
Not yet measured
Pending Phase B validation.
Not yet measured
Pending Phase B training runs.
110 + 5
v0.3.1
Limitations
- Real benchmark results are pending Linux Phase B — the current contribution is the benchmark foundation, not final empirical conclusions.
- The failure taxonomy has its infrastructure in place ahead of full per-environment instrumentation.
- G1 hand open/close action signs still need in-sim verification.
- The motion-planning demonstration path awaits a Linux mplib shakedown.
- A diffusion-policy baseline is a stretch goal, not a current result.
Lessons & tradeoffs
- Benchmarks earn trust through their splits: leakage-proof paraphrase and object holdouts were designed in before any training, so later claims can't quietly overfit.
- Rehearsing the full reporting pipeline on labeled synthetic data separates infrastructure bugs from science before any GPU-hours are spent.
Artifacts
- Code (private — public release pending)not yet published
- Benchmark reportnot yet published
- Artifact pack (real Phase B results)not yet published
- Rollout videosnot yet published