World Model Atlas

Evaluation framework · Published 2026-07-10

How do you evaluate a world model?

A model is not useful merely because its video looks plausible. Evaluation must begin with the route, the decision it supports and the evidence available outside the demo.

Evidence: mixed primary and independent sources Claim type: bounded editorial judgment Last reviewed: 2026-08-08

Short answer / 一句话结论

First ask what the model predicts and who acts on the prediction. Then test nine checks across three layers: capability (state, control, action consequence, long horizon and fidelity), task utility (closed loop, downstream transfer and calibration), and evidence quality (reproducibility and access). Weight the checks by route. A video generator, a latent planning model and a driving simulator should not share one leaderboard or an additive total score.

The evaluation matrix / 评估矩阵

Start at the decision, not the demo. / 从要支持的决策开始，而不是从演示开始。

  1. 01 · ScopeRoute / 路线What is predicted, represented or generated?
  2. 02 · CapabilityModel / 模型What state and dynamics does it preserve?
  3. 03 · UtilityTask / 任务Does it improve the downstream decision?
  4. 04 · EvidenceAccess / 证据Can outsiders inspect or reproduce the result?
  5. 05 · DecideDecision / 判断Is it useful under stated limits and missing evidence?

Evaluation framework

1. Define the route before choosing a metric

“World model” covers systems with different outputs: pixels, 3D scenes, latent states, future sensor observations or representations used by a controller. The 2018 World Models project evaluated a compact learned dynamics model through agent behavior. Genie 3 emphasizes navigable generated environments, V-JEPA 2 predictive representations, and GAIA-2 action-conditioned driving futures. The same metric cannot express success across all four.

2. Audit three layers across nine checks

Capability / What the model can preserve and predict

CheckQuestion to testUseful evidenceCommon false positive
StateDoes the representation preserve objects, relations and variables the task needs?State probes, object tracking, geometry or task-state recovery.Visually plausible frames with hidden state loss.
ControlCan prompts, actions or conditions change the intended variable without uncontrolled changes?Intervention tests, controllability and disentanglement.A prompt appears to work on selected examples.
Action consequenceDoes the future respond correctly to the agent's action?Action-conditioned prediction, inverse dynamics and counterfactual checks.Passive continuation mistaken for interaction — or a model that does respond, and still shows the task succeeding when the action should have made it fail.
Long horizonDo identity, geometry, goals and causal effects survive repeated rollout?Error curves by horizon, re-entry tests and multi-step task completion.A short clip hides compounding drift.
FidelityAre dynamics and frequencies faithful, not merely images realistic?Physical constraints, distributional calibration and measured-world comparison.High perceptual quality treated as physics.

Task utility / What changes for a downstream user

CheckQuestion to testUseful evidenceCommon false positive
Closed loopCan a policy act, receive the changed state and act again?Policy-in-the-loop evaluation and adversarial interventions.Open-loop replay called a simulator.
Downstream transferDoes the model improve planning, control, learning or data efficiency?Task success, regret, sample efficiency and robust transfer.A representation benchmark with no decision benefit.
CalibrationDoes uncertainty rise when the model is wrong or outside its data?Reliability curves, OOD detection and risk-coverage tests.Confident output with unknown failure probability.

Evidence quality / What outsiders can verify

CheckQuestion to testUseful evidenceCommon false positive
Reproducibility and accessCan outsiders inspect the setup, data split, metric and failure cases?Open protocol, code/model access, repeated runs and independent tests.A company-selected demo with no test protocol.

Do not add these checks into one score. Capability, task utility and evidence quality answer different questions. A strong result in one layer cannot compensate for missing evidence in another.

Two of these checks can also pull against each other, which is different from saying they do not substitute for each other. MiraBench, a robotic-manipulation benchmark posted in May 2026, perturbs an action so that the task must fail — too little grip force, an early release, a slipped carry — and then asks whether the predicted video still shows the task succeeding. On its reported figures, post-training one model for task success raised its completion score from 14.0 to 92.0 while its failure-preservation score fell from 48.7 to 23.1. Two boundaries belong with that pair of numbers. The post-trained model is the benchmark authors' own fine-tune of a third-party base, so it evidences that recipe rather than success-oriented post-training in general; and at the larger 14B scale the same comparison moves not at all, both models sitting at 12.8. The underlying failure — a model reproducing the outcome its training data made common rather than the one the input specified — is the same mechanism this site has already described for driving simulators as distribution bias. What is new is that it is being measured on the action-outcome axis rather than described.

There is also a check the nine above do not contain: all of them ask about the model, and none asks about the instrument that produced the score. MiraBench is worth reading on this because it publishes enough to be checked. Its optimism-bias level is scored by one vision-language model used zero-shot, and its appendix states that two different prompts were used — a standard one for action-conditioned models and a deliberately lenient one for text-conditioned models, whose rendering style differs. In the table validating that judge against human annotators, the single model scored with the lenient prompt is also the one where agreement collapses: it recovers 33.3% of the cases humans marked as biased, against 78.3% to 100% for every model scored with the standard prompt. Reported agreement overall is 87.8%. None of this is concealed — the prompts, the per-model agreement and the full configuration are printed — and an under-detecting judge inflates a reliability score rather than deflating it. A position paper from a separate group makes the general point without reference to this benchmark: model-as-judge evaluation is useful and scalable, but a judge can reward a plausible-looking outcome while missing the action-relevant error, so it should not stand as ground truth for decision usefulness. When a score comes from a judge, ask what validated the judge, on which models, and whether it was applied the same way to all of them.

3. Weight the matrix by route

RouteHighest-weight checksUseful metricsEvidence boundary
Video or world generationState, control, long horizon, physical fidelity.Object persistence, geometry, intervention success and drift by duration.Perceptual scores alone do not prove interaction or planning utility.
Interactive generated worldControl, action consequence, closed loop, long horizon.Input latency, action responsiveness, state consistency and policy success.A navigable demo does not establish stable agent training.
Predictive representationState, downstream transfer, calibration.Probe quality, task transfer, data efficiency and OOD behavior.Strong representation transfer is not full scene simulation.
Robotics or drivingAction consequence, closed loop, fidelity, calibration, transfer.Trajectory error, intervention validity, policy regret, rare-case coverage and real-system transfer.Simulation results remain one layer of a deployment case.

4. Separate three evidence layers

A

Developer evidence

Official releases and technical reports establish what was tested, under the developer's protocol. They are primary evidence for the claim, not independent confirmation.

B

Independent validation

External reproduction, downstream use and reality-linked tests establish whether results survive outside the originating team.

C

Editorial judgment

Synthesis can compare routes and identify missing evidence, but it must not upgrade an official claim into an industry fact.

This article is layer C. It synthesizes public primary and technical sources already reviewed by World Model Atlas. It does not introduce a universal benchmark, and it does not claim independent reproduction of the named systems.

5. Use the checklist before accepting “world model” performance

  1. What exactly is predicted: pixels, state, geometry, reward, action or a representation?
  2. What is the operational task, and which failure would matter?
  3. Is the test open loop or policy in the loop?
  4. How does error change with rollout horizon?
  5. Are actions interventions or only labels correlated with the future?
  6. Is fidelity measured against reality, or only against human preference?
  7. Does the model improve a downstream decision over a credible baseline?
  8. Are uncertainty and out-of-distribution behavior reported?
  9. Can an independent team inspect or reproduce the result?
  10. Which conclusion remains unsupported after all reported metrics?

Editorial conclusion

A credible world model is not the model with the highest single visual score. It is the model whose representation and dynamics remain useful under the interventions, horizons and downstream decisions its route requires, with uncertainty and access limits stated. Until those pieces are visible, “world model” is a research direction or product framing, not a validated capability class.

Technical evidence records / 技术证据记录

Evidence table / 证据表

Claim ID / 判断 IDClaim / 判断Sources / 来源Confidence and boundary / 置信度与边界
claim-world-model-evaluation-multidimensional Useful world-model evaluation is multidimensional; no single visual metric establishes planning, control or deployment utility. World Models, Genie 3, V-JEPA 2, GAIA-2 Medium-high as editorial synthesis; no universal independent benchmark is claimed.
claim-world-model-evaluation-route-specific Evaluation weights should follow the model route and downstream decision rather than one shared leaderboard. Genie 3, Cosmos 3, V-JEPA 2, GAIA-2 Medium-high; this is a taxonomy judgment, not a standardized scoring protocol.
claim-world-model-evaluation-closed-loop Passive visual quality cannot establish action-conditioned or closed-loop usefulness. World Models, GAIA-2, Cosmos 3GAIA-2 Sec. 2.2, p. 6 and Fig. 7, p. 13; Cosmos 3 Table 18, p. 65 and Fig. 22, p. 66. Medium-high for the distinction; public evidence remains system- and protocol-specific.
claim-world-model-evaluation-deployment-boundary In safety-critical deployment, world-model benchmarks are only one evidence layer and do not constitute a system-level safety case. RAND, readable PDF p. 10, Koopman & WagnerRAND discussion, p. 10; Koopman & Wagner testing discussion, p. 6 and conclusion, p. 9. High for autonomous-driving validation; medium when generalized beyond safety-critical systems.
claim-world-model-evaluation-evaluator-is-audited When a benchmark score is produced by an automated judge, the judge is part of what has to be audited: MiraBench scores optimism bias with one VLM but two prompts split by model type, and the one model given the lenient prompt is the one whose agreement with human annotators collapses. MiraBench, decision-centric evaluation positionMiraBench Appendix D.2.4, pp. 27–28 and Table 10, p. 28; scores stated as evaluator output on p. 8. Position paper Sec. 7.1, p. 21. Medium-high, bounded to MiraBench as published on 28 May 2026. The benchmark discloses the prompt split and the per-model agreement itself, and its overall reported agreement is 87.8%; the position paper is independent of it but does not mention it.

Primary and boundary sources / 主要与边界来源

  1. Ha & Schmidhuber — World ModelsLearned dynamics evaluated through an agent and control task.
  2. Google DeepMind — Genie 3Official framing and public capability limits for interactive generated worlds.
  3. Meta AI — V-JEPA 2Official predictive-representation and downstream physical-reasoning route.
  4. Wayve — GAIA-2 technical reportDriving-domain action-conditioned generation and scenario controls.
  5. NVIDIA — Cosmos 3 technical reportPhysical-AI evaluations involving action and inverse dynamics.
  6. RAND — Driving to SafetyIndependent boundary on statistical road validation and simulation's supplementary role.
  7. Koopman & Wagner — Autonomous Vehicle SafetyIndependent boundary on testing and system-level safety evidence.
  8. MiraBench (arXiv:2605.29360v1)Action-conditioned failure-preservation protocol and its automated-evaluator configuration. Unrefereed preprint; its authors fine-tuned five of the twelve models it scores.
  9. Decision-making-centric world-model evaluation (arXiv:2606.15032v2)Independent boundary on treating model-as-judge output as evidence of decision usefulness. Unrefereed preprint; reports no experiments of its own.

Source access and review notes are maintained in Sources and sources.json.

Why autonomous driving needs world models Genie, Cosmos and V-JEPA: three routes What is a world model? Sources