The robot already had the fan guard in its fingers. It only had to line it up and push. It pushed. The fan slid away.

That episode comes from a Claude Opus 5.5 run documented in Embodied First Exam. The project’s newer leaderboard gives Claude Fable 5.2 roughly 80% when it can read the simulator and start over. Restrict the same agent to the standard interface on matched tasks and the score drops below half.

Count the same tasks twice

The project lists 1,156 tasks across 35 suites, 23 simulators and 47 robots. We used the task-level records available on November 19 to compare the 391 core tasks that offer both modes, and counted the full set of 456 standard core tasks separately.

Claude Fable 5.2 in Claude Code, high reasoning effort, 391 matched tasks
OutcomeTasksWhat it captures
Passed both175Agreement across interfaces
Passed privileged only137About 35.0% of the paired set
Passed standard only9Individual attempts can go either way
Failed both70Full state and retries were insufficient

The totals are 312/391 (79.8%) with privileges and 184/391 (47.1%) in standard mode, a gap of about 32.7 percentage points. The reference report counted 311 privileged passes; the project’s task-level data available for our check counted 312. On all 456 standard core tasks, Fable passed 226 (49.6%), GPT‑6 Astra in Codex passed 157 (34.4%), and GPT‑6.1 Sol passed 147 (32.2%). The public arXiv v1 covers an earlier eight-agent cohort; it is the source for the protocol and ablation below.

Embodied First Exam · Official leaderboard · Embodied First Exam · Leaderboard data · Embodied First Exam · arXiv v1, Sections 2–3 · Embodied First Exam · Per-task results · 新智元 · Embodied First Exam report

The interface changes the job

Agents may issue actions directly, write a controller or train a policy within their budget. A verifier outside the agent scores the attempt. The leaderboard therefore measures a model working through a specific harness and interface.

What the agent gets to see and recover
SettingObservationsWhat happens after a mistake
PrivilegedFull simulator state, including ground-truth object posesUnlimited resets; a submitted trajectory is verified
StandardSensor observations and robot state, depending on the suiteNo reset by default; recover from the current state
RealtimeStandard interface while the environment keeps movingSlow planning can close the control window

A paused scene gives the agent time to estimate a cup’s position. A moving ball gives it a deadline. When a system misses that deadline, the tail of inference latency, stale frames and controller behavior between actions matter more than a neat average.

Embodied First Exam · Official leaderboard · Embodied First Exam · arXiv v1, Sections 2–3 · Embodied First Exam · Public task runner

The last few degrees

In Figure 12, Opus 5.5 holds the guard 36–39 degrees off-axis in standard mode. The fit allows about five degrees. Pushing harder displaces the fan. A single label of “assembly failed” would hide whether the agent misread the pose, issued the wrong correction or could no longer recover.

The v1 paper tested GPT‑6 Astra in Codex by restoring capabilities one at a time. It reported 28% at baseline, 45% with exact object state, 53% with resets and saved states, and 68% with both. That experiment helps explain why a reset can be valuable. The Fable leaderboard comparison does not isolate the share due to either treatment.

An exact pose can improve the initial approach. A reset can let the agent revise a grasp after contact. Record those capabilities separately, especially when a task has tight tolerances and the first error physically changes the scene.

Embodied First Exam · arXiv v1, Sections 2–3

Turn the scoreboard into a rerunnable test

Start with a safe simulated tabletop task, such as placing a soft block in a tray at a specified angle. Freeze the model, harness version, observation interface, budget and verifier. Use the same initial scenes across conditions and hold some scenes out.

  1. Run sensor-only, exact-state, reset-enabled and both-enabled variants. Repeat each condition and retain unsuccessful trajectories.
  2. Label approach, contact, completion and verification. Record the first irrecoverable mistake and count successful recoveries.
  3. For time-sensitive tasks, run a separate realtime condition. Publish the numerator and denominator, P95 latency, timeouts, collisions and human interventions. Check whether the pattern survives on held-out scenes.

A better score can justify a larger experiment. Before trying the result on hardware, examine which failures survive tighter permissions, where recovery breaks and which interventions an operator would actually be able to make.

Embodied First Exam · arXiv v1, Sections 2–3 · Embodied First Exam · Public task runner