What have world models actually achieved in robotics?

English

What counts as a robotics world model here?

This article includes a system only when it learns action-conditioned predictive states or dynamics and uses those predictions for planning, control or policy learning. A visual encoder, a robot policy or a generated robot video is not enough by itself. The included systems still differ: Dreamer learns a task policy through imagined latent rollouts; visual foresight plans through predicted images; V-JEPA 2-AC optimizes actions against predicted representations.

1. Real-world online learning: DayDreamer

DayDreamer is direct evidence that a learned latent world model can support reinforcement learning on physical hardware rather than only in simulation. The authors used the same Dreamer algorithm and main hyperparameters across four robot-task setups, but each setup was trained separately. This was not one shared model transferring zero-shot between robots.

• Unitree A1: one reported training run learned to roll over, stand and walk in about one hour; after pushing began, it adapted for roughly ten more minutes.
• UR5: visual pick-and-place reached about 2.5 objects per minute after eight hours.
• XArm: reached about 3.1 objects per minute after ten hours; the soft object was attached to the gripper with a string to reduce corner failures.
• Sphero: learned navigation to a fixed goal in about two hours with a dense distance reward.

The boundary matters as much as the times. The arm tasks used explicit rewards, the “human performance” reference came from three people operating the UR5 for 20 minutes each, and the authors warn that hours of hardware learning create wear and may require intervention or repair. DayDreamer supports physical feasibility and sample reuse—not reward-free learning, transfer without retraining or unattended deployment.

2. Image-goal planning in new labs: V-JEPA 2-AC

V-JEPA 2 first pretrains a video representation model, then freezes its encoder and trains an action-conditioned predictor. The post-training subset contains under 62 hours from DROID: short Franka videos plus end-effector state. “Unlabeled” here means no task type, reward or success label; actions are still constructed from adjacent end-effector states.

The authors deployed the same weights on Franka arms in two labs that were not part of DROID. In Table 2 on PDF page 14, each lab/task/object cell contains ten trials. Averaged across the two labs, the paper reports 100% reach; grasp at 65% for a cup and 25% for a box; reach-with-object at 75% for both; and pick-and-place at 80% for a cup and 65% for a box.

These are meaningful closed-loop results, but “zero-shot” has a local meaning: the deployment labs supplied no training data and the system received no task-specific finetuning there. It does not mean no robot interaction data. The hardware remained Franka + RobotiQ; the team manually tried camera positions before choosing one that worked; and pick-and-place used multiple human-provided image subgoals. The paper also identifies autoregressive error accumulation and growing action-search space as long-horizon limits.

3. Cross-robot data helps, but adaptation is not zero-shot

RoboNet studies predictive control across data from multiple robots. Its strongest held-out-robot result is not direct transfer. The target robot contributes 300–400 trajectories for finetuning.

The authors caution that tasks differed across robots and human judges determined success. The tasks were mostly low-fidelity pushing and pick-and-place. They also report underfitting even with much larger video-prediction models. The result is best read as a data-efficiency finding inside a narrow protocol.

4. What controlled benchmarks can—and cannot—show

DINO-WM and TD-MPC2 make the algorithmic case clearer. DINO-WM learns patch-feature dynamics from offline trajectories and plans toward observation goals in six controlled benchmark environments. TD-MPC2 evaluates latent model-predictive control across 104 tasks in four benchmark domains; its 80-task model uses 50 Meta-World and 30 DMControl tasks, 545 million replay-buffer transitions and scales to 317 million parameters.

Those experiments support planning and control under their protocols. They are not physical-robot deployments. Task count is not a count of real-world abilities, and benchmark performance does not establish hardware robustness or safety.

5. Reproduction changes the picture without erasing the original result

A 2026 TMLR paper, What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?, retrained V-JEPA 2-AC and reported that the public implementation's two-step rollout loss did not match the formula in the paper. After fixing it, the authors retrained the model. They also found that planner choice could materially change results in their benchmarks.

This is valuable code-level reproduction, not a rerun of V-JEPA 2 Table 2. Its DROID evaluation used offline action scores on 16 collected Franka videos, and RoboCasa was simulated. Bardes and LeCun overlap with the original V-JEPA line, so the evidence is best classified as cross-protocol reproduction with author overlap.

A second study, stable-worldmodel-v1, reimplemented DINO-WM on Push-T. It reports 94% success for goals drawn from expert trajectories, 12% for random-policy trajectory goals, and 4%–20% under tested unseen visual or physical variations. Its planning budget also differs from the original protocol, and Yann LeCun is again an overlapping author. This is a useful robustness result, not independent real-robot validation.

The evidence state

In our bounded search through public paper, web and code indexes up to 12 July 2026, we did not find a third-party reproduction of these systems on the same physical-robot tasks and protocols. We did find the two author-overlap reproductions above. “Not found in this search” is not “does not exist,” and lack of independent reproduction is not evidence that the original capability is false.

The most defensible summary is therefore: mechanisms work, tasks remain limited, and validation is still thin. Robotics world models have moved beyond simulation-only claims. They have not yet earned the stronger label of general physical intelligence.

中文

本文如何界定“机器人世界模型”？

只有当一个系统学习动作条件下的状态或动力学预测，并把预测用于规划、控制或策略学习时，本文才将其纳入。单独的视觉编码器、机器人策略或机器人视频生成并不够。纳入的系统也不是同一种算法：Dreamer 在潜空间想象中学习任务策略，Visual Foresight 通过预测图像规划，V-JEPA 2-AC 则在预测表征中优化动作。

1. 直接在真实世界学习：DayDreamer

DayDreamer 提供了直接证据，表明潜空间世界模型可以在实体硬件上支持强化学习，而不只存在于模拟器。作者在四组机器人与任务中使用相同的 Dreamer 算法和主要超参数，但每组任务分别训练；这不是同一个共享权重模型在不同机器人之间零样本迁移。

• Unitree A1：论文报告的一次训练约一小时学会翻身、站立和行走；开始施加推动后又适应约十分钟。
• UR5：视觉抓放训练约八小时后达到每分钟约 2.5 个物体。
• XArm：约十小时达到每分钟约 3.1 个物体；软物体通过绳子连接夹爪，以降低卡在角落的概率。
• Sphero：使用稠密距离奖励，约两小时学会导航到固定目标。

这些条件和时间同样重要。机械臂任务使用明确奖励；“接近人类表现”的参照来自三位操作者各用摇杆控制 UR5 20 分钟；作者也指出，数小时硬件学习会造成磨损，可能需要人工干预或维修。DayDreamer 支持的是实体可行性和样本利用，而不是无需奖励、无需重新训练的跨机器人迁移或无人值守部署。

2. 在新实验室做图像目标规划：V-JEPA 2-AC

V-JEPA 2 先预训练视频表征模型，再冻结编码器并训练动作条件预测器。后训练数据来自不足 62 小时的 DROID 子集：短时 Franka 视频和末端执行器状态。“无标签”是指没有任务类型、奖励或成功标签；动作仍由相邻末端状态构造。

作者把相同权重部署到两个没有进入 DROID 的实验室。在 PDF 第 14 页 Table 2 中，每个实验室、任务和物体组合有十次试验。两实验室平均结果为：Reach 100%；杯子/盒子 Grasp 为 65%/25%；持物移动为 75%/75%；抓取放置为 80%/65%。

这是有意义的闭环结果，但“零样本”只在局部成立：部署实验室没有提供训练数据，也没有在当地做任务微调。它不等于没有机器人交互数据。硬件仍是 Franka + RobotiQ；团队手工尝试不同相机位置后才选定可工作设置；抓取放置还使用多个人工图像子目标。论文也明确指出，长时规划会受到自回归误差累积和动作搜索空间增长限制。

3. 多机器人数据有帮助，但适配不是零样本

RoboNet 研究了多机器人数据上的预测控制。其 held-out robot 结果不是直接迁移，而是使用目标机器人的 300–400 条轨迹微调。

作者提醒，不同机器人实验难度不同，成功由人工判断；任务主要是低精度推物和抓取放置。论文还报告，即使更大的视频预测模型也出现欠拟合。因此最稳妥的说法是：这是特定协议中的数据效率结果。

4. 受控 benchmark 能证明什么？

DINO-WM 和 TD-MPC2 更清楚地展示了算法机制。DINO-WM 从离线轨迹学习 patch-feature 动力学，在六个受控 benchmark 环境中按观察目标规划。TD-MPC2 在四个 benchmark 域的 104 项任务上评估潜空间模型预测控制；其 80 任务模型由 50 个 Meta-World 和 30 个 DMControl 任务组成，使用 5.45 亿条 replay-buffer transitions，最大模型为 3.17 亿参数。

这些实验支持协议内的规划与控制，但不是实体机器人部署。任务数不等于现实能力数，benchmark 成绩也不能证明硬件鲁棒性或安全性。

5. 复现改变了证据结构，但没有抹掉原始结果

2026 年 TMLR 论文 What Drives Success in Physical Planning with Joint-Embedding Predictive World Models? 重训了 V-JEPA 2-AC，并报告公开实现中的两步 rollout loss 与论文公式不一致。修复后作者重新训练模型，也发现 planner 选择会明显改变其 benchmark 结果。

这是有价值的代码级复现，但没有复跑 V-JEPA 2 Table 2。其 DROID 评估是在 16 条自采 Franka 视频上计算离线 action score，RoboCasa 是模拟器；Bardes 和 LeCun 与原 V-JEPA 路线有作者重合。因此更准确的分类是“带作者重合的跨协议复现”。

另一项研究 stable-worldmodel-v1 在 Push-T 上重新实现 DINO-WM：expert trajectory 目标为 94%，random-policy trajectory 目标降至 12%，测试的未见视觉或物理变化为 4%–20%。其规划预算和原协议不同，Yann LeCun 也有作者重合。这是鲁棒性边界，不是独立实体机器人验证。

当前证据状态

在截至 2026 年 7 月 12 日、限定于公开论文、网页和代码索引的检索中，我们没有找到这些系统在相同实体任务和协议下完成的第三方独立复现；找到的是上述两项带作者重合的后续复现。“本轮未找到”不等于“不存在”，缺少独立复现也不等于原始能力为假。

因此最稳妥的总结是：机制已经可行，任务仍然有限，验证依然偏薄。机器人世界模型已经越过“只在模拟器里”的阶段，但还没有获得“通用物理智能”的证据。

Evidence map / 证据地图

Sources

• DayDreamer: World Models for Physical Robot Learning — physical robot experiments.
• V-JEPA 2 — action-conditioned post-training and two-lab deployment.
• DROID v2 — robot data provenance; its arXiv metadata says 84 tasks while the v2 PDF says 86.
• RoboNet and Visual Foresight — predictive control and robot adaptation lineage.
• DINO-WM and TD-MPC2 — controlled benchmark evidence.
• What Drives Success in Physical Planning with JEPA-WMs? and stable-worldmodel-v1 — bounded reproduction and robustness evidence.