Physical AI Data Flywheel
A closed-loop data-to-policy system that turns robot benchmark failures into targeted demonstration requests — then validates, trains, evaluates, and repeats.
Role — Creator — system design, schemas, validation, selection strategies, artifact pipeline
- Robot-learning pipelines usually treat data collection as a fixed upfront cost: gather demonstrations, train, hope. But when a policy fails, those failures say exactly where the dataset is thin. The flywheel asks whether closing that loop — failures deciding what data comes next — beats collecting more of the same.
- Data-to-policy loop · dataset validation & selection infrastructure
- Data quality and data choice are becoming the bottleneck of physical AI. A trustworthy flywheel needs validation, provenance, and honest accounting built in from the start — otherwise 'more data' quietly becomes 'more of the same failure modes'.
- Solo project, built as a layer inside the x-embodiment benchmark codebase.
Overview
Robot demonstrations are expensive, so which ones you collect matters. Instead of gathering more data at random, this flywheel — built inside the cross-embodiment benchmark — reads evaluation failures, generates candidate demonstration requests aimed at what actually failed, selects among them under an episode budget, validates the resulting dataset before any training touches it, and packages every round into a reproducible artifact with dataset cards and provenance. The Mac-safe foundation is complete: seventeen modules, typed schemas end to end, deterministic validation, five selection strategies, and a full toy round rehearsed on synthetic data with explicit warnings. The real collect–train–evaluate loop is pending Linux experiments, which are designed to test one falsifiable question: does failure-targeted data improve performance more efficiently than the same budget of random additional demonstrations?
Evaluation episodes stream failure records into analysis; failures become candidate demonstration requests; a selection strategy spends a fixed episode budget across candidates; collected demonstrations pass deterministic validation before entering the training set; the retrained policy is re-evaluated, and the whole round — dataset card, validation report, plans, and comparisons — is packaged as a checksummed artifact. Then the loop runs again.
- Evaluation failures → failure analysis
- Candidate demonstration requests
- Budgeted selection — 5 strategies
- Collection → dataset validation
- Policy training → re-evaluation
- Round artifact pack — cards · provenance
Contributions
- Designed the round loop: evaluation failures → failure analysis → candidate demonstration requests → budgeted selection → collection → dataset validation → training → re-evaluation → artifact packs.
- Built seventeen flywheel modules with typed schemas for provenance, dataset index records, quality issues, candidate demo requests, selection plans, dataset cards, model-card placeholders, and round manifests.
- Implemented deterministic dataset validation with quality scoring — datasets are checked before any training consumes them.
- Implemented five budgeted selection strategies — random, stratified, failure_targeted, diversity_greedy, and cost_aware_failure_targeted — so the core comparison (targeted vs. random) is a config switch, not a rewrite.
- Rehearsed the entire loop as a deterministic toy round producing synthetic artifact packs with explicit warnings, and generated Linux command plans without executing any training on the Mac.
- Covered the layer with 77 flywheel tests; the repo-wide suite stands at 187 fast + 5 simulation tests.
Evidence & evaluation
Evidence
Flywheel foundation
attached17 modules, typed schemas end to end, deterministic validation with quality scoring, and failure-driven candidate generation.
Selection strategies
attachedrandom · stratified · failure_targeted · diversity_greedy · cost_aware_failure_targeted — all budgeted, all tested.
Toy round rehearsal
attachedDeterministic end-to-end round on synthetic data, packaged into artifact packs with explicit synthetic-data warnings.
Test suite
attached77 flywheel tests; repo-wide suite at 187 fast + 5 simulation tests, all passing.
Real flywheel rounds
pendingThe collect–train–evaluate loop on real data awaits Linux experiments; all current outputs are synthetic/toy.
Targeted-vs-random experiment
pendingPre-registered comparison: baseline vs. +random demos vs. +failure-targeted demos at equal budget.
Metrics
Not yet measured
The flywheel's headline metric; pending real Linux rounds.
Not yet measured
Equal-budget comparison; pending real experiments.
Not yet measured
Computed by validation on real collected data; pending.
17
77
Limitations
- Mac-safe foundation complete; the real data/training/evaluation loop is pending Linux experiments — no flywheel results exist yet.
- All current flywheel outputs are synthetic/toy and are explicitly labeled as such unless marked real later.
- Failure targeting currently works at the granularity of the benchmark's existing failure labels; sharper targeting lands with richer per-episode instrumentation.
- Designed and rehearsed on one benchmark (xembench); generality beyond it is a hypothesis, not a claim.
Lessons & tradeoffs
- Validation belongs before training, not after a bad run: a dataset the pipeline can't trust is a result you can't trust.
- Making 'which data next?' a typed, budgeted, testable decision turns a vague intuition — failures should guide collection — into an experiment with a falsifiable answer.
Artifacts
- Code (private — public release pending)not yet published
- Flywheel experiment reportnot yet published
- First real round artifact packnot yet published