Do VLMs have internal World Models?
Towards an Atomic Evaluation

Qiyue Gao*, Xinyu Pi*, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai,
Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen,
David Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu
UCSD, JHU, Cornell Tech, EPFL, UMich
ACL 2025
*Indicates Equal Contribution
arXiv 🤗 WM-ABench @ Maitrix.org Discord Discord

We present WM-ABench, a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations.

Abstract

Internal world models (WMs) enable agents to understand the world's state and predict transitions, serving as the basis for advanced deliberative reasoning. Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs. While the latest studies have evaluated and shown limitations in specific capabilities such as visual understanding, a systematic evaluation of VLMs' fundamental WM abilities remains absent. Drawing on comparative psychology and cognitive science, we propose a two-stage framework that assesses perception (visual, spatial, temporal, quantitative, and motion) and prediction (mechanistic simulation, transitive inference, compositional inference) to provide an atomic evaluation of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations. Through 660 experiments on 15 latest commercial and open-source VLMs, we find that these models exhibit striking limitations in basic world modeling abilities. For instance, almost all models perform at near-random accuracy when distinguishing motion trajectories. Additionally, they lack disentangled understanding---e.g., some models tend to believe blue objects move faster than green ones. More rich results and analyses reveal significant gaps between VLMs and human-level world modeling.

Benchmark Overview

WM-ABench Model Performance

Benchmark Leaderboard

Perception

Prediction

* Frontier VLMs evaluated on subset of WM-ABench.

Dataset Visualization

Key Findings

Finding 1: VLMs tend to entangle irrelevant physical concepts (Figure 1).

When we change one dimension, like color, while keeping others constant, we find that models' perception across almost all dimensions will be affected, such as the object's size and speed. Systematic evaluation shows that color and shape emerge as the most influential attributes across the tasks.

Relative Entanglement Scores by Tasks
Figure 1: Heatmap showing s-RE scores that quantify the relative impact of different physical dimensions on perception task performance. Higher values (darker red) indicate stronger entanglement between a physical dimension and task performance, while lower values (lighter colors) suggest weaker relationships.

Finding 2: Accurate perception does not guarantee accurate prediction (Figure 2).

Even when models perfectly perceive the current state, they still fail at predicting physical outcomes on three tasks. This suggests that perception errors aren't the sole cause of poor prediction. Models lack the fundamental physical knowledge necessary to simulate object interactions accurately.

Figure 2: Model performance between filtered (correct state perception) and unfiltered inputs

BibTeX

@misc{gao2025visionlanguagemodelsinternalworld,
  title={Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation},
  author={Qiyue Gao and Xinyu Pi and Kevin Liu and Junrong Chen and Ruolan Yang and Xinqi Huang and Xinyu Fang and Lu Sun and Gautham Kishore and Bo Ai and Stone Tao and Mengyang Liu and Jiaxi Yang and Chao-Jung Lai and Chuanyang Jin and Jiannan Xiang and Benhao Huang and Zeming Chen and David Danks and Hao Su and Tianmin Shu and Ziqiao Ma and Lianhui Qin and Zhiting Hu},
  year={2025},
  eprint={2506.21876},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2506.21876}
}