A Chinese report dated November 14 drew attention to Fable 5.2 topping a new RoboColiseum leaderboard. We checked the original public leaderboard data, current technical report and code documentation. The central news claim holds: in our snapshot, Fable 5.2 has a Season II Overall score of 18.30 and ranks first both overall and among Agents. This is a result for a particular evaluation system. “First in robotics” would extend it beyond the tasks, deployment conditions and capabilities actually measured.
RoboColiseum original leaderboard · RoboColiseum current technical report · BAAI Chinese news republication
We first align names and dates, then reconstruct the weighting and examine execution frameworks and budgets. Finally, we provide an offline audit that calls no model and connects to no robot. The result tables reproduce public scores; the alternative weighting, recording contract and acceptance process are our analytical designs. We have not rerun the leaderboard, tested physical robots or independently authenticated each submission.
Align the benchmark, model release and evidence versions
RoboColiseum Season II, RoboColosseum and RoboDojo are different evaluations. Their similar names do not make tasks, scores or training conditions interchangeable. Anthropic released Fable 5.2 on November 9. Its launch article’s RoboDojo Sim result of 64.9% comes from a different internal evaluation at the highest available effort; it is not the 18.30 score examined here.
RoboColiseum original leaderboard · RoboColiseum current technical report · October 20 Agent2Act preprint, version 1 · Anthropic Fable 5.2 / Mythos 5.2 announcement · RoboColosseum distributed physical-evaluation preprint · BAAI Chinese news republication
Current leaderboard entries contain submission-time strings dated November 13 without a stated timezone. They cannot automatically be converted into a Singapore publication time. November 14 is the date displayed on the Chinese republication we read, not proof of when the original board first went live. We use a November 15 access snapshot; the original site does not provide enough publication history to establish its first release time.
The October 20 Agent2Act preprint is an earlier cross-model evaluation and does not include Fable 5.2. The current new report is titled “RoboColiseum: Co-evolution of Simulation, Evaluation, and Models”; its report index offered no available arXiv link when checked. Cite the earlier paper, new web report and live leaderboard separately rather than using the older paper as reproducibility evidence for a new result.
| Name | Use in this article | Keep separate |
|---|---|---|
| RoboColiseum Season II | Main board here: four simulated tracks | 18.30 is a score on a 0–100 scale |
| RoboColosseum | A separate distributed physical-robot evaluation | Do not import its trials or rankings |
| RoboDojo Sim | Another simulation evaluation in Anthropic’s launch | 64.9% is from vendor-run internal tests |
RoboColiseum original leaderboard · Anthropic Fable 5.2 / Mythos 5.2 announcement · RoboColosseum distributed physical-evaluation preprint
Read the four tracks alongside the overall ranking
The table shows the current top five systems overall, using the site’s 0–100 display scale. Fable is particularly strong in Precision Manipulation and Cross-Embodiment Transfer. Its Whole-Body Coordination score is below Astra and Opus, while its Dexterous Manipulation score is below Gemini and Opus. Overall leadership describes this aggregation rule, not a win in every track.
RoboColiseum original leaderboard · RoboColiseum current technical report
| System | Category | Overall | Precision | Cross-embodiment | Whole-body | Dexterity |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 | Agent | 18.30 | 33.04 | 17.99 | 18.42 | 3.73 |
| GPT-6 Astra | Agent | 13.73 | 23.60 | 12.37 | 18.96 | 0.00 |
| Claude Opus 5.5 | Agent | 11.77 | 14.18 | 9.25 | 18.94 | 4.70 |
| π0.5 | Policy | 11.22 | 4.20 | 10.00 | 13.87 | 16.79 |
| Gemini 4 Argon | Agent | 10.74 | 11.49 | 8.86 | 17.42 | 5.19 |
RoboColiseum original leaderboard · RoboColiseum current technical report
π0.5 has a lower overall score than Fable but scores 16.79 on dexterity, above the general-purpose models shown. The report’s Policy systems receive mixed-task supervised fine-tuning with 200 demonstrations per task. Agents and Policies differ in adaptation, training and inference. Use the combined view to identify profiles, then compare within categories and examine costs separately.
Equal track weights are different from equal task weights
The official Overall score is the arithmetic mean of four track scores. Fable’s displayed values give (33.04 + 17.99 + 18.42 + 3.73) / 4 = 18.295, which rounds to 18.30. Its raw leaderboard total is 0.18295, displayed on a 0–100 scale. Decimal storage or percentage-like formatting does not turn the metric into a proportion of successful attempts.
RoboColiseum original leaderboard · RoboColiseum current technical report · RoboColiseum official usage and scoring rules
Coverage is unequal: Cross-Embodiment Transfer evaluates six task families across four embodiments, making 24 cases; the other tracks contain eight, eight and ten tasks. Cross-embodiment accounts for 48% of these 50 cases but still receives 25% of the four-track macro-average. This favors balanced capability dimensions. Researchers and buyers should ask whether those dimensions match their workload.
| Track | Task structure | Case count | Overall weight |
|---|---|---|---|
| Precision Manipulation | 8 tasks | 8 | 25% |
| Cross-Embodiment Transfer | 6 task families × 4 embodiments | 24 | 25% |
| Whole-Body Coordination | 8 tasks | 8 | 25% |
| Dexterous Manipulation | 10 tasks | 10 | 25% |
RoboColiseum current technical report · RoboColiseum official usage and scoring rules
The example below reweights the published track aggregates by task counts, producing 17.6148. It is a mathematical sensitivity exercise, not an official task-level micro-average or another measurement. Four aggregates do not retain every within-track scoring and sample weight. The program uses only the Python standard library, and we validated its arithmetic offline.
from decimal import Decimal, ROUND_HALF_UP
# Published rounded track scores; arithmetic only, no model or simulator call.
tracks = {
"precision": Decimal("33.04"),
"cross_embodiment": Decimal("17.99"),
"whole_body": Decimal("18.42"),
"dexterous": Decimal("3.73"),
}
macro = sum(tracks.values()) / Decimal(4)
print("macro_from_displayed_tracks:", macro) # 18.295
print("display:", macro.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP))
# Counterfactual weighting exercise, NOT an official micro-average or a rerun.
counts = {"precision": 8, "cross_embodiment": 24, "whole_body": 8, "dexterous": 10}
weighted = sum(tracks[k] * counts[k] for k in tracks) / sum(counts.values())
print("counterfactual_weighting:", weighted) # 17.6148
assert sum(counts.values()) == 50
assert macro == Decimal("18.295")
assert weighted == Decimal("17.6148")Progress, completion rewards and budgets all affect the score
The rules award progress from task state and predefined subgoals, with sequential or parallel milestones depending on the task. Timeout, leaving the permitted area or dropping an object can end an episode with unfinished components uncredited. Finishing the entire task within budget earns an additional reward. Partial progress, complete success and the final score therefore need separate reporting.
RoboColiseum current technical report · RoboColiseum official usage and scoring rules
Task execution-step limits are derived from training data. Agents also face an inference-call budget, with deductions for exceeding a reasonable call allowance. Additional retries, longer action sequences or extra feedback can change the capability-and-resource combination being measured. More reasoning is an experimental condition with a cost, rather than a detail to ignore.
The leaderboard and overview we examined do not disclose all penalty coefficients, completion-bonus values and per-run logs needed to reconstruct each row. We do not invent a complete scoring formula. Request independent fields for raw progress, completion reward and budget deductions, then reconcile the total; reverse-engineering a success rate from the total is unreliable.
The evaluated unit is a model plus its execution framework
Agent2Act connects a multimodal model’s instruction, visual observations and robot state to action conversion, local validation and execution feedback. The model proposes targets; the execution layer handles coordinate conversion, inverse kinematics or trajectory compilation. Rejected actions lead to bounded replanning. A leaderboard row therefore measures the model together with observation interfaces, tools and a control layer.
Agent2Act official code and README · RoboColiseum official platform repository
The current README distinguishes proposed actions, issued commands and observed outcomes. Producing executable commands does not establish successful manipulation. It also says the example G2 configuration disables speed validation, so that example does not demonstrate velocity or dynamics compliance. This limits the example; without evidence, it does not establish that a leaderboard run used the same configuration.
The platform repository separates hosted evaluation from local execution: local results do not automatically enter the online board. The following is our proposed audit record, not an official response schema. null means evidence is still required. In particular, the repository’s current HEAD must not be silently substituted for the commit actually used in a November 13 run.
{
"record_type": "proposed_audit_contract_not_official_result",
"board": "RoboColiseum Season II",
"accessed_date": "2026-11-15",
"model_display_name": "Claude Fable 5.2",
"model_snapshot_id": null,
"harness_commit": null,
"simulator_commit": null,
"split_hash": null,
"task_id": null,
"embodiment": null,
"layout_id": null,
"seed": null,
"effort": null,
"temperature": null,
"observation_contract": null,
"control_step_limit": null,
"inference_call_limit": null,
"control_steps_used": null,
"inference_calls_used": null,
"partial_score": null,
"completion_bonus": null,
"budget_penalty": null,
"end_to_end_success": null,
"wall_time_seconds": null,
"cost_usd": null,
"termination_reason": null,
"artifact_sha256": null
}What hidden layouts, dynamic sets and human review address
The official protocol says public development layouts and server test layouts use different instances from the same distribution. The server also maintains a fixed official set and a dynamic set that changes per submission. Dynamic scores do not enter the total; discrepancies from the official set help assess overfitting risk. Human review adds checks for privileged information and techniques that affect evaluation outcomes.
RoboColiseum current technical report · RoboColiseum official usage and scoring rules · OpenRUA analysis of robot-agent evaluation
These mechanisms improve comparability. The existence of a hidden set does not prove the absence of every leak or form of overfitting. Record whether tuning touched test tasks, whether skills were repeatedly optimized on those tasks and whether the agent received live completion judgments unavailable in deployment. Protocol declarations and evidence of an individual submission’s compliance are distinct materials.
OpenRUA’s November 2 methodological analysis examines unequal budgets, leaked completion judgments and skills optimized for test tasks. It concerns other robot-agent experiments and general protocol risks. We use it to formulate review questions, not to transfer an allegation of misconduct to RoboColiseum or Fable without evidence.
Keep uncertainty visible when samples and intervals are missing
The public aggregates do not establish all repeat counts, per-task variances or confidence intervals for Fable’s row. The report mentions 240 robot–task cases for cross-embodiment, but that cannot be extrapolated to a rollout total for every track or used to fill in matching seeds and independent sample counts for each model.
RoboColiseum original leaderboard · RoboColiseum current technical report · RoboColiseum official usage and scoring rules
For example, Astra scores 18.96 and Opus 18.94 on Whole-Body Coordination, a displayed gap of just 0.02 points. Without run-level data, stability is unknown; decimal places are not measurement precision. Repeats within the same task or embodiment can be correlated. Treating every frame as an independent sample would create unjustified certainty.
For your own evaluation, freeze tasks, layouts and budgets in advance and pair systems on matching conditions. Keep each result, estimate uncertainty at an appropriate task or layout level, and report failures, timeouts and service errors separately. An interval crossing zero means the evidence is insufficient for that difference. Zero complete successes must remain visible rather than disappearing behind partial-progress filtering.
| Question | Freeze or record | Acceptance evidence |
|---|---|---|
| Is the model stronger? | Matching tasks/layouts, observations, control layer, effort and budgets | Paired outcomes, intervals and complete failure logs |
| Is it economical? | Actual calls, tokens, wall time, retries and charges | Cost per complete success and tail latency |
| Does it generalize across embodiments? | Embodiments held out from adaptation, task split and coordinate interfaces | Per-embodiment results alongside aggregates |
| Is physical deployment justified? | Hardware, cameras, calibration, action limits and takeover | Independent low-risk trials, emergency stops and recovery records |
Real2Sim’s 0.9686 is a correlation coefficient
The new report compares 22 paired robot–task configurations spanning 16 tasks and reports Pearson r = 0.9686 between real and simulated task scores. Each pair uses that task’s own Score definition. This is not 96.86% correct actions, 96.86% successful tasks or a 96.86% simulation-to-reality agreement rate.
Correlation measures how quantities vary together; it does not guarantee equal values. A simulator can systematically overestimate physical performance while retaining high correlation. Inspect the scatterplot, equality line, fitted line, paired differences and calibration errors together. When task score scales differ, also examine which between-task differences drive the correlation.
These results motivate further study of the relationship between simulation rankings and physical performance. They do not validate Fable 5.2 on your hardware, cameras, friction, latency or safety conditions. Physical deployment needs independent low-risk trials, action boundaries and human takeover. We performed none of those validations.
Turn leaderboard reading into a reproducible evidence audit
Give the following prompt to a research assistant with downloaded report, rule and result snapshots. Ask for a claim–source locator–calculation–unknowns audit rather than another promotional summary assembled from the web. Do not provide account keys, personal identifiers or unauthorized robot-control interfaces.
RoboColiseum original leaderboard · RoboColiseum current technical report · RoboColiseum official usage and scoring rules · Agent2Act official code and README · RoboColiseum official platform repository
Role: robotics-evaluation evidence auditor. Use only the public snapshots I supply.
Task: audit the Fable 5.2 result in RoboColiseum Season II.
1. Separate the original board, rules, new web report, earlier paper and media.
2. List Overall and four track scores; recalculate the equal-weight mean.
Do not relabel the score as a success rate.
3. Locate model snapshot, effort, budgets, repeats, seeds and intervals.
4. Write unknown for missing fields. Do not fill historical settings from current HEAD.
5. Separate Real2Sim correlation, calibration and this model's physical acceptance.
Output: claim | source_locator | calculation | limitation.
End with three minimal evidence requests. Do not allege cheating or claim replication.The audit artifact should include access date, source-file hashes, season and category filters, track scores, formulas and missing fields. Compare changes before updating a conclusion when a page changes. A screenshot alone is insufficient for arithmetic reconstruction; current code alone is insufficient to establish a historical run configuration.
Move from a leaderboard signal to deployment evidence
- Confirm name, season, category and access date
- Save original values and rule versions
- Recompute weights and inspect track-level counterexamples
- Obtain model, framework, budget and sample records
- Run your own paired tasks and failure analysis
- Validate physical deployment independently
Plan the next model evaluation from these results
Fable 5.2’s lead is a useful candidate-selection signal. The evaluated system’s precision and cross-embodiment scores justify prioritizing investigations into planning, visual targeting and constrained correction. Explaining the cause still requires control of execution layers, observations, budgets and repeats; replacing a model label alone does not isolate model capability.
RoboColiseum original leaderboard · RoboColiseum current technical report · Anthropic Fable 5.2 / Mythos 5.2 announcement
For multi-finger contact tasks, start with dexterity and failure logs and compare specialized Policies. For mobile manipulation, examine whole-body coordination and safety limits. If your work is documents, code or research, a robotics leaderboard is an indirect signal. Build separate tests around your own inputs, outputs and acceptance criteria.
The most useful outcome is a plan for the next experiment: what capability deserves testing, what evidence is missing and what physical risks remain uncovered. A research ranking should inform deployment only after workload-matched success, cost, latency and failure-recovery evidence are available together.
This article audits the interpretability of public evidence. Only the arithmetic example was executed offline. We did not rerun a robotics leaderboard, confirm every submission’s internal configuration or obtain a new physical success rate for Fable 5.2. On the next update, repeating this evidence chain is more useful than remembering a champion’s name.