Learning Transferable Dynamics Priors from Action to World Modeling

Ze Huang^{1 *}, Jiahui Zhang^{1 *}, Hairuo Liu^{2 3 *}, Chenxi Zhang², Ran Cheng⁴, Li Zhang^{1 2 †}

¹Fudan University  ²Shanghai Innovation Institute  ³Shanghai Jiao Tong University  ⁴McGill University

^*Equal contribution  ^†Corresponding author

We view action-conditioned world modeling as a transferable dynamics prior for robot learning. A2World is pretrained on 2.1M+ robot manipulation trajectories spanning 20+ embodiments to predict future multi-view manipulation videos from an initial observation and future action chunks. The same pretrained dynamics prior can be adapted into A2World-sim, a long-horizon autoregressive simulator for policy evaluation, and A2World-policy, a video-action joint prediction model for instruction-conditioned robot control.

Abstract

We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict how actions drive visual scene evolution, the resulting world model captures reusable interaction dynamics beyond appearance-level video generation.

Concretely, we pretrain a multi-view interactive base diffusion world model, A2World, on large-scale robot manipulation data with real action annotations. We validate the learned dynamics priors from two complementary perspectives. First, we adapt A2World into a task- or scene-specialized real-world simulator, A2World-sim, whose long-horizon rollouts support simulator-based policy evaluation and scalable what-if analysis by replacing real-robot rollouts with world model rollouts. Second, starting from the same pretrained weights, we adapt A2World into a video-action joint prediction model, A2World-policy, that predicts actions under visual and instruction conditioning.

Experiments across simulation benchmarks and real-robot settings demonstrate that action-conditioned world model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.

Method

A2World learns action-to-video dynamics from robot data, then transfers the pretrained prior into two downstream variants:

• A2World-sim injects pose-guided history and rolls out future observations autoregressively for long-horizon simulator-based evaluation.
• A2World-policy performs joint video-action diffusion with shared attention and action-specific denoising branches for instruction-conditioned control.

Real-Robot Execution Videos

A2World-policy is evaluated on a Flexiv dual-arm real-robot suite covering precision insertion, reorientation, switch interaction, lifting, and deformable-object handling.

put-chain-in-box.mp4 Put chain in the box
toggle-power-switch.mp4 Toggle power switch	flip-small-box.mp4 Flip small box
insert-ram-module.mp4 Insert RAM module	lift-box-high.mp4 Lift box high

BibTeX

If you find this project helpful, please consider citing our paper:

@article{huang2026a2world,
    title={Learning Transferable Dynamics Priors from Action to World Modeling},
    author={Huang, Ze and Zhang, Jiahui and Liu, Hairuo and Zhang, Chenxi and Cheng, Ran and Zhang, Li},
    year={2026},
    journal={arXiv preprint},
    eprint={2606.29501},
    archivePrefix={arXiv},
}