The robot already had the fan guard in its fingers. It only had to line it up and push. It pushed. The fan slid away.
That episode comes from a Claude Opus 5.5 run documented in Embodied First Exam. The project’s newer leaderboard gives Claude Fable 5.2 roughly 80% when it can read the simulator and start over. Restrict the same agent to the standard interface on matched tasks and the score drops below half.
Count the same tasks twice
The project lists 1,156 tasks across 35 suites, 23 simulators and 47 robots. We used the task-level records available on November 19 to compare the 391 core tasks that offer both modes, and counted the full set of 456 standard core tasks separately.
| Outcome | Tasks | What it captures |
|---|---|---|
| Passed both | 175 | Agreement across interfaces |
| Passed privileged only | 137 | About 35.0% of the paired set |
| Passed standard only | 9 | Individual attempts can go either way |
| Failed both | 70 | Full state and retries were insufficient |
The totals are 312/391 (79.8%) with privileges and 184/391 (47.1%) in standard mode, a gap of about 32.7 percentage points. The reference report counted 311 privileged passes; the project’s task-level data available for our check counted 312. On all 456 standard core tasks, Fable passed 226 (49.6%), GPT‑6 Astra in Codex passed 157 (34.4%), and GPT‑6.1 Sol passed 147 (32.2%). The public arXiv v1 covers an earlier eight-agent cohort; it is the source for the protocol and ablation below.
Embodied First Exam · Official leaderboard · Embodied First Exam · Leaderboard data · Embodied First Exam · arXiv v1, Sections 2–3 · Embodied First Exam · Per-task results · 新智元 · Embodied First Exam report
The interface changes the job
Agents may issue actions directly, write a controller or train a policy within their budget. A verifier outside the agent scores the attempt. The leaderboard therefore measures a model working through a specific harness and interface.
| Setting | Observations | What happens after a mistake |
|---|---|---|
| Privileged | Full simulator state, including ground-truth object poses | Unlimited resets; a submitted trajectory is verified |
| Standard | Sensor observations and robot state, depending on the suite | No reset by default; recover from the current state |
| Realtime | Standard interface while the environment keeps moving | Slow planning can close the control window |
A paused scene gives the agent time to estimate a cup’s position. A moving ball gives it a deadline. When a system misses that deadline, the tail of inference latency, stale frames and controller behavior between actions matter more than a neat average.
Embodied First Exam · Official leaderboard · Embodied First Exam · arXiv v1, Sections 2–3 · Embodied First Exam · Public task runner
The last few degrees
In Figure 12, Opus 5.5 holds the guard 36–39 degrees off-axis in standard mode. The fit allows about five degrees. Pushing harder displaces the fan. A single label of “assembly failed” would hide whether the agent misread the pose, issued the wrong correction or could no longer recover.
The v1 paper tested GPT‑6 Astra in Codex by restoring capabilities one at a time. It reported 28% at baseline, 45% with exact object state, 53% with resets and saved states, and 68% with both. That experiment helps explain why a reset can be valuable. The Fable leaderboard comparison does not isolate the share due to either treatment.
An exact pose can improve the initial approach. A reset can let the agent revise a grasp after contact. Record those capabilities separately, especially when a task has tight tolerances and the first error physically changes the scene.
Turn the scoreboard into a rerunnable test
Start with a safe simulated tabletop task, such as placing a soft block in a tray at a specified angle. Freeze the model, harness version, observation interface, budget and verifier. Use the same initial scenes across conditions and hold some scenes out.
- Run sensor-only, exact-state, reset-enabled and both-enabled variants. Repeat each condition and retain unsuccessful trajectories.
- Label approach, contact, completion and verification. Record the first irrecoverable mistake and count successful recoveries.
- For time-sensitive tasks, run a separate realtime condition. Publish the numerator and denominator, P95 latency, timeouts, collisions and human interventions. Check whether the pattern survives on held-out scenes.
A better score can justify a larger experiment. Before trying the result on hardware, examine which failures survive tighter permissions, where recovery breaks and which interventions an operator would actually be able to make.
Embodied First Exam · arXiv v1, Sections 2–3 · Embodied First Exam · Public task runner