Affordance-guided data collection
Intervene when the scene can no longer support valid interaction — not merely when a task fails.
BEDA — Bidirectional Evaluation of Data Affordance
Large-scale robotic learning increasingly relies on diverse real-world interaction data, yet collecting such data remains difficult to scale. Although autonomous collection systems reduce the need for continuous low-level teleoperation, inherent failures and environment degradation still require human intervention when continued execution becomes infeasible. This shifts the bottleneck from controlling the robot to deciding when the environment requires a manual reset. We propose an affordance-guided robotic data collection system that maintains productive scene states by using data affordance: whether the current scene can support the continued collection of a valid interaction dataset. At its core, BEDA (Bidirectional Evaluation of Data Affordance) decomposes scene diagnosis into retrospective evaluation of trajectory validity and prospective evaluation of next-task feasibility. A task-agnostic diagnosis protocol is instantiated with task descriptions to generate task-specific checklists, enabling structured VLM-based evaluation of whether collection should continue or request intervention. Across 15 real-world tabletop tasks, our affordance-guided collection system reaches 84.3% of the throughput upper bound. Scaled multi-robot simulations further show increased operator capacity, with a single operator maintaining 91% of ideal throughput across fleets of up to 128 robots. BEDA's intervention decisions closely align with human oracle labels, and ablations confirm that both retrospective and prospective evaluations are necessary. These results establish affordance-guided intervention as a practical mechanism for scalable long-horizon robotic data collection.
Intervene when the scene can no longer support valid interaction — not merely when a task fails.
Bidirectional evaluation turns open-ended failure judgment into two structured checks.
Looks backward at the episode that just finished: was the collected trajectory a valid piece of interaction data? A nominally successful rollout can still be invalid, and a failed grasp can still leave usable data behind.
Looks forward at the resulting scene: can the next task still be executed from here? Displaced objects, fallen items, or a jammed drawer make continued collection infeasible even after a success.
15 real-world manipulation tasks, one hour of autonomous collection per task suite, repeated nine times per strategy. Human interventions happen only when the strategy asks for one.
| Intervention strategy | Data throughput ↑ | Execution success ↑ (%) | Data affordance ↑ | Intervention freq. (/hr) |
|---|---|---|---|---|
| RADAR | 0.149 ±0.048 | 16.8 ±5.6 | 0.178 ±0.038 | – |
| Unstructured VLM | 0.619 ±0.116 | 60.4 ±11.4 | 0.615 ±0.106 | 11.56 ±7.53 |
| Oracle-Period | 0.696 ±0.085 | 63.9 ±4.4 | 0.772 ±0.070 | 8.80 ±1.44 |
| BEDA (ours) | 0.843 ±0.030 | 82.8 ±4.7 | 0.874 ±0.051 | 7.87 ±2.09 |
A behavior cloning policy trained on the collected demonstrations improves from 38.3% held-out success at N = 50 to 67.5% at N = 300. Under a matched collection-time budget, the policy trained on BEDA data reaches 3.1× lower normalized k-step action prediction error than the reverse transfer.
Each bidirectional evaluation takes 3.75 s and about 2,100 tokens per episode. The reported throughput already includes this latency, the reset decision, and manual reset time — so the VLM evaluation is not an added bottleneck in practice.
Removing either half of the bidirectional evaluation costs throughput.
| Variant | Throughput ↑ | Exec. success ↑ (%) | Affordance ↑ | Interv. freq. (/hr) |
|---|---|---|---|---|
| Prospective-only | 0.387 | 23.7 | 0.233 | 3.04 |
| Retrospective-only | 0.656 | 58.3 | 0.621 | 26.20 |
| BEDA (both) | 0.843 | 82.8 | 0.874 | 7.87 |
Intervention decisions compared against human data affordance labels.
Errors concentrate in fine-grained scene assessment — object–object spatial relations judged from correct detections — and in omitted functional state, such as whether a drawer is jammed.
Reset-needed states are re-evaluated after every episode, so 92.4% of missed-detection streaks resolve within two evaluations, and invalid demonstrations make up only 2.0% of the accepted data.
A human-in-the-loop simulation where a single operator oversees parallel robot streams, each becoming reset-needed with probability 0.1 after an episode.
BEDA's decision outcomes are replayed from its measured real-robot decision profile, while evaluator and operator confirmation latencies are measured online.
As the fleet grows, continuous supervision becomes bottlenecked by episode-wise checks, and periodic intervention stays limited by delayed detection. Allocating operator attention by affordance keeps throughput near ideal with low robot waiting.
15 real-world manipulation tasks grouped into four transitionable task pools. Each pool pairs a forward task with the task that returns the scene, so collection can continue without a human. Three pools are shown below.
A tabletop workspace with an AgileX PiPER arm and three RGB cameras. BEDA is instantiated with GPT-5.4 as the VLM backbone and a fixed intervention threshold of 0.9 across every task.
