X-Embodiment Language-Grounded Manipulation Benchmark
A two-embodiment ManiSkill3 benchmark whose first large campaign found near-zero transfer under weak precision policies — then measured where action chunking helped and where it did not.
- Cross-embodiment claims are hard to interpret when tasks, action spaces, language splits, and native policy quality all change at once.
- Simulation-only benchmark + baseline campaign + intervention study
- A useful benchmark should expose when transfer is impossible to interpret, not turn a weak native policy into a misleading transfer claim.
Who did what
- Project owner — research question, benchmark architecture, experimental decisions, campaign supervision, analysis, and artifact review
- Independent research project; AI coding agents assisted implementation and campaign operations under explicit review and verification.
- ManiSkill3 environments and robot assets, PyTorch, Franka Panda and Unitree G1 simulation models.
Overview
Xembench asks a narrow question: how much performance survives when the task and language stay fixed but the robot body changes? A shared language-conditioned policy emits actions through a canonical 21-dimensional interface, and thin adapters map those actions to a Franka-style arm or Unitree G1 upper body. The completed Phase B baseline used 1,202 demonstrations, 18 behavior-cloning policies, and a 73-cell / 6,550-episode matrix. Push-T reached 45.7% native success and G1 box transport reached 18.8%; zero-shot cross-embodiment transfer was approximately zero because each transfer pair included a weak native side. A controlled follow-up found a real positive boundary: action chunking raised tool-pulling success from 1.3% to 14.7% across three seeds, while grasp-critical tasks stayed near zero.
Methodology
- Represented both embodiments through a canonical 21-dimensional semantic action interface and thin robot-specific adapters.
- Trained language-conditioned behavior-cloning baselines from collected demonstrations, then evaluated native, held-out-language, unseen-color, and zero-shot body-swap cells through one reporting pipeline.
- Used a failure taxonomy to choose an equal-budget targeted-versus-random data round and a later intervention ladder rather than treating more data as the automatic remedy.
Language-conditioned observations enter one policy stack. The policy emits a canonical action vector whose semantic slots remain fixed across bodies; a thin adapter maps valid slots to each robot. Native, held-out-language, unseen-color, and zero-shot body-swap cells share one evaluation and reporting pipeline.
- Language-grounded task
- Shared policy stack
- Canonical 21-D action
- Franka / G1 adapter
- Native + transfer evaluation matrix
My contribution
- Defined a canonical 21-dimensional semantic action interface and per-robot adapters so a zero-shot body swap is an explicit, testable operation.
- Built leak-checked paraphrase and unseen-color splits, a demonstration pipeline, behavior-cloning and PPO baselines, evaluation matrices, and failure-taxonomy reports.
- Executed and reviewed the 1,202-demonstration, 6,550-episode Phase B campaign and stopped short of claiming transfer when native precision policies were too weak.
- Ran one real failure-driven data-flywheel round with equal targeted and random budgets; the mostly null result showed that more same-kind data was not the binding constraint.
- Designed a precision-manipulation intervention ladder and confirmed across three seeds that action chunking rescued tool pulling but not grasp-critical tasks.
Provenance & claim boundary
- The project uses ManiSkill3 environments and robot assets; it does not claim authorship of the simulator or underlying robot models.
- AI coding agents assisted substantial implementation and campaign operations. The public claim is project direction, experiment design, run supervision, analysis, and verification — not sole manual authorship of every file.
- All results are simulation-only. No sim-to-real or hardware-transfer claim is made.
Experimental design
- The completed Phase B campaign used 1,202 demonstrations, 18 behavior-cloning policies, and 73 evaluation cells totaling 6,550 episodes.
- Native competence was checked before interpreting cross-embodiment transfer; transfer pairs containing a weak native side were not treated as clean embodiment tests.
- The action-chunking comparison was repeated across three seeds and reported separately for tool pulling and grasp-critical tasks.
Results & evidence
Evidence
Baseline campaign
attached1,202 demonstrations, 18 BC policies, 73 cells, and 6,550 evaluation episodes.
Native baselines
attachedPush-T 45.7%; G1 box transport 18.8%; precision grasping near zero.
Intervention result
attachedAction chunking improved pull_tool from 1.3% to 14.7% across three seeds; pick/stack remained near zero.
Reproducibility pack
attachedConfigs, checkpoints, CSV matrices, reports, driver scripts, checksums, and artifact manifests are linked in the repository.
Metrics
6,550 episodes
1,202
45.7% / 18.8%
1.3% → 14.7%
Failure analysis
- Zero-shot transfer was approximately zero, but the binding ambiguity was weak native precision manipulation rather than embodiment alone.
- The equal-budget failure-driven data round was mostly null, indicating that more same-kind demonstrations did not resolve the observed ceiling at this scale.
- Action chunking improved temporally extended tool pulling while leaving pick-and-stack tasks near zero, localizing a persistent last-inch grasp-conversion problem.
Limitations
- Cross-embodiment transfer was near zero and cannot be interpreted as a clean embodiment effect while native precision policies remain weak.
- The benchmark is simulation-only and uses a deliberately constrained policy class; results do not generalize to stronger sequence models or physical robots.
- The first failure-driven data round was underpowered for near-zero tasks and produced a mostly null comparison.
Lessons & tradeoffs
- Native competence is a prerequisite for interpreting transfer.
- Longer action horizons can help temporally extended tool use without solving last-inch grasp conversion.
- A null result can narrow the search space: at this scale, more same-kind demonstrations were not enough.
Next questions
- Which policy classes can first establish reliable native precision manipulation on both embodiments?
- How should action representations separate shared task semantics from embodiment-specific control authority?
- Once native competence is established, which task factors produce genuine zero-shot or adaptation-based transfer?