World Model Atlas

Evaluation framework · Published 2026-07-10

How do you evaluate a world model?

A model is not useful merely because its video looks plausible. Evaluation must begin with the route, the decision it supports and the evidence available outside the demo.

Evidence: mixed primary and independent sources Claim type: bounded editorial judgment Last reviewed: 2026-07-10

Short answer / 一句话结论

First ask what the model predicts and who acts on the prediction. Then test nine dimensions: state, control, action consequence, long horizon, fidelity, closed loop, downstream transfer, calibration and reproducibility. Weight them by route. A video generator, a latent planning model and a driving simulator should not share one leaderboard.

The evaluation scorecard / 评估计分卡

Start at the decision, not the demo. / 从要支持的决策开始，而不是从演示开始。

  1. 01 · ScopeRoute / 路线What is predicted, represented or generated?
  2. 02 · TestDimensions / 维度Which of the nine capabilities matter for this use?
  3. 03 · WeightTask / 任务Planning, generation and validation need different weights.
  4. 04 · VerifyEvidence / 证据Separate developer results from independent checks.
  5. 05 · DecideUtility / 效用Does it improve a downstream decision under known limits?

Evaluation framework

1. Define the route before choosing a metric

“World model” covers systems with different outputs: pixels, 3D scenes, latent states, future sensor observations or representations used by a controller. The 2018 World Models project evaluated a compact learned dynamics model through agent behavior. Genie 3 emphasizes navigable generated environments, V-JEPA 2 predictive representations, and GAIA-2 action-conditioned driving futures. The same metric cannot express success across all four.

2. Audit nine dimensions

DimensionQuestion to testUseful evidenceCommon false positive
StateDoes the representation preserve objects, relations and variables the task needs?State probes, object tracking, geometry or task-state recovery.Visually plausible frames with hidden state loss.
ControlCan prompts, actions or conditions change the intended variable without uncontrolled changes?Intervention tests, controllability and disentanglement.A prompt appears to work on selected examples.
Action consequenceDoes the future respond correctly to the agent's action?Action-conditioned prediction, inverse dynamics and counterfactual checks.Passive continuation mistaken for interaction.
Long horizonDo identity, geometry, goals and causal effects survive repeated rollout?Error curves by horizon, re-entry tests and multi-step task completion.A short clip hides compounding drift.
FidelityAre dynamics and frequencies faithful, not merely images realistic?Physical constraints, distributional calibration and measured-world comparison.High perceptual quality treated as physics.
Closed loopCan a policy act, receive the changed state and act again?Policy-in-the-loop evaluation and adversarial interventions.Open-loop replay called a simulator.
Downstream transferDoes the model improve planning, control, learning or data efficiency?Task success, regret, sample efficiency and robust transfer.A representation benchmark with no decision benefit.
CalibrationDoes uncertainty rise when the model is wrong or outside its data?Reliability curves, OOD detection and risk-coverage tests.Confident output with unknown failure probability.
Reproducibility and accessCan outsiders inspect the setup, data split, metric and failure cases?Open protocol, code/model access, repeated runs and independent tests.A company-selected demo with no test protocol.

3. Weight the matrix by route

RouteHighest-weight dimensionsUseful metricsEvidence boundary
Video or world generationState, control, long horizon, physical fidelity.Object persistence, geometry, intervention success and drift by duration.Perceptual scores alone do not prove interaction or planning utility.
Interactive generated worldControl, action consequence, closed loop, long horizon.Input latency, action responsiveness, state consistency and policy success.A navigable demo does not establish stable agent training.
Predictive representationState, downstream transfer, calibration.Probe quality, task transfer, data efficiency and OOD behavior.Strong representation transfer is not full scene simulation.
Robotics or drivingAction consequence, closed loop, fidelity, calibration, transfer.Trajectory error, intervention validity, policy regret, rare-case coverage and real-system transfer.Simulation results remain one layer of a deployment case.

4. Separate three evidence layers

A

Developer evidence

Official releases and technical reports establish what was tested, under the developer's protocol. They are primary evidence for the claim, not independent confirmation.

B

Independent validation

External reproduction, downstream use and reality-linked tests establish whether results survive outside the originating team.

C

Editorial judgment

Synthesis can compare routes and identify missing evidence, but it must not upgrade an official claim into an industry fact.

This article is layer C. It synthesizes public primary and technical sources already reviewed by World Model Atlas. It does not introduce a universal benchmark, and it does not claim independent reproduction of the named systems.

5. Use the checklist before accepting “world model” performance

  1. What exactly is predicted: pixels, state, geometry, reward, action or a representation?
  2. What is the operational task, and which failure would matter?
  3. Is the test open loop or policy in the loop?
  4. How does error change with rollout horizon?
  5. Are actions interventions or only labels correlated with the future?
  6. Is fidelity measured against reality, or only against human preference?
  7. Does the model improve a downstream decision over a credible baseline?
  8. Are uncertainty and out-of-distribution behavior reported?
  9. Can an independent team inspect or reproduce the result?
  10. Which conclusion remains unsupported after all reported metrics?

Editorial conclusion

A credible world model is not the model with the highest single visual score. It is the model whose representation and dynamics remain useful under the interventions, horizons and downstream decisions its route requires, with uncertainty and access limits stated. Until those pieces are visible, “world model” is a research direction or product framing, not a validated capability class.

Evidence table / 证据表

Claim ID / 判断 IDClaim / 判断Sources / 来源Confidence and boundary / 置信度与边界
claim-world-model-evaluation-multidimensional Useful world-model evaluation is multidimensional; no single visual metric establishes planning, control or deployment utility. World Models, Genie 3, V-JEPA 2, GAIA-2 Medium-high as editorial synthesis; no universal independent benchmark is claimed.
claim-world-model-evaluation-route-specific Evaluation weights should follow the model route and downstream decision rather than one shared leaderboard. Genie 3, Cosmos 3, V-JEPA 2, GAIA-2 Medium-high; this is a taxonomy judgment, not a standardized scoring protocol.
claim-world-model-evaluation-closed-loop Passive visual quality cannot establish action-conditioned or closed-loop usefulness. World Models, GAIA-2, Cosmos 3GAIA-2 Sec. 2.2, p. 6 and Fig. 7, p. 13; Cosmos 3 Table 18, p. 65 and Fig. 22, p. 66. Medium-high for the distinction; public evidence remains system- and protocol-specific.
claim-world-model-evaluation-deployment-boundary In safety-critical deployment, world-model benchmarks are only one evidence layer and do not constitute a system-level safety case. RAND, readable PDF p. 10, Koopman & WagnerRAND discussion, p. 10; Koopman & Wagner testing discussion, p. 6 and conclusion, p. 9. High for autonomous-driving validation; medium when generalized beyond safety-critical systems.

Primary and boundary sources / 主要与边界来源

  1. Ha & Schmidhuber — World ModelsLearned dynamics evaluated through an agent and control task.
  2. Google DeepMind — Genie 3Official framing and public capability limits for interactive generated worlds.
  3. Meta AI — V-JEPA 2Official predictive-representation and downstream physical-reasoning route.
  4. Wayve — GAIA-2 technical reportDriving-domain action-conditioned generation and scenario controls.
  5. NVIDIA — Cosmos 3 technical reportPhysical-AI evaluations involving action and inverse dynamics.
  6. RAND — Driving to SafetyIndependent boundary on statistical road validation and simulation's supplementary role.
  7. Koopman & Wagner — Autonomous Vehicle SafetyIndependent boundary on testing and system-level safety evidence.

Source access and review notes are maintained in Sources and sources.json.

Why autonomous driving needs world models Genie, Cosmos and V-JEPA: three routes What is a world model? Sources