3D Point World Models

Abstract

Large video-based dynamics models lack explicit 3D structure and suffer from geometrically inconsistent long-term rollouts. Emerging 3D dynamics models based on partial point clouds are more consistent but remain sensitive to occlusions and accumulated drift. We present 3D Point World Models (3DPWM) – a task-agnostic world model that operates entirely in 3D space by first completing partial point clouds and then learning action-conditioned dynamics in the completed scene. Operating on completed geometry enables reliable long-horizon rollouts and more accurate cost evaluation for model-based planning, while supporting adaptation to new tasks. Across embodiments and tabletop manipulation benchmarks, 3DPWM achieves substantially higher rollout accuracy than state-of-the-art 3D world models including PointWorld, supports open-loop and closed-loop planning, and enables sim-to-real transfer.

Key takeaways

Better long-horizon rollouts

Predicted point clouds achieve strong reward correlations (0.77–0.86) across all six tasks, while baselines reach at most 0.45 on long-horizon tasks.

Closed-loop planning

Closed-loop planning achieves 2–4× higher success than ParticleFormer+.

Open-loop planning

Open-loop planning achieves up to 6× higher success with learned completion than prior rollout-based baselines.

Adaptation to unseen long-horizon tasks

Trained only on MugCleanup and Coffee demonstrations, 3DPWM adapts to the unseen long-horizon CoffeePreparation task, where ParticleFormer+ fails.

Sim-to-real transfer

Trained in simulation only and deployed on a Franka with a single RGB-D camera and no real-world fine-tuning, 3DPWM does about 2.25× better than baselines.

Real-world videos

Franka Panda, single static RGB-D camera, MPPI planning on 3DPWM rollouts. The dynamics model is the best PickYCB simulation checkpoint, with no real-world fine-tuning. Videos are silent.