arxiv:2610.21095

Can World and Action Modeling Benefit Each Other under Unified Causal Training?

Published on Oct 15

· Submitted by

Selen Su on Oct 16

Interactive and Active AI

Upvote

Authors:

Yue Su ,

Kuan Zhang ,

Abstract

OneWM unifies causal training of world and action modeling, showing that learning action-conditioned dynamics improves policies and action learning improves multi-view world prediction.

Generated by thinkingmachines/Inkling-Small

World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.

View arXiv page View PDF Project page GitHub 71 Add to collection

Community

selensu

Paper author Paper submitter 3 days ago

•

edited 1 day ago

OneWM — official resources

OneWM in 27 seconds

A 1080p overview: a complete real-robot stacking task (6×); a selected Cream cheese world-prediction clip with ground-truth comparison, recorded end-effector trajectory, and input actions (0.5×, replayed); and the first 16 seconds of the two-object Endless Dream (2×), with synchronized predicted controls. The dream control path accumulates translation commands within each round; it is not a measured robot trajectory.

Real-world robot demonstrations

Full robot recordings at original speed, without audio.

Block stacking — stacking three colored blocks.

Storage — inserting a blue block into a storage unit.

Breakfast preparation — plating toast and a fried egg.

An Endless Dream

An imagined rollout starting from initial scene and wrist-camera observations. OneWM alternates action prediction and future-frame generation, using its own predictions to continue the rollout. Frames after the initial observations are model-generated; the predicted actions are not executed in a simulator.

Alphabet soup into the basket

Two objects into the basket

The first 16 seconds at original speed, with synchronized scene and wrist views plus predicted actions: the relative control path, translation, rotation, and gripper values. The path accumulates saved translation commands within each round; it is not a measured robot trajectory.