Mundo, developed in collaboration with the S-Lab at Nanyang Technological University, turns images into interactive 3D environments. The world model combines physics simulation and video generation to train robots to navigate and manipulate objects.
Presented as a new robotics simulation tool, Mundo can reconstruct places, modify their contents programmatically and compute the physical consequences of actions. The demonstrations described include scenes that can be explored from different angles, objects subjected to forces and robots carrying out virtual tasks. The goal is to have customised environments in which robot behaviours can be prepared and analysed before they are applied in real locations.
How Mundo builds 3D environments from 1 to 12 images
In the demonstrations presented, Mundo uses between 1 and 12 images to reconstruct rooms and make them explorable virtually. The system analyses the layout of the furniture, the dimensions of objects and the spatial relationships between the different views.
A 3D detector produces an initial representation of the scene. An agent based on a vision-language model then adjusts the positions and dimensions of the elements by comparing the renders with the images provided.
When the virtual camera moves, Mundo preserves the visual constraints of the photographed areas and generates the parts that do not appear in the original images. The system can also extend the visual world of an illustration or a generated image.
For larger spaces, Mundo uses several dozen context images. It selects the references suited to each section of the path and stitches the sequences together while maintaining a shared 3D representation. According to the capabilities presented, this method produces continuous explorations lasting from several tens of seconds to several minutes.
Physics engine: how Mundo simulates motion and collisions
Mundo does more than produce images of virtual environments. Its architecture combines a 3D representation, scene programs, a physics engine and a video generation model.
The programs describe the objects present, their dimensions, positions and orientations. They can also define physical properties such as mass and friction.
Scripts then determine the actions to perform and the forces to apply. The physics engine computes movements, contacts and collisions, taking gravity into account among other factors. The video model generates the images that correspond to the computed results.
The interactions presented include several operations:
- Pushing or pulling objects in a virtual scene.
- Opening drawers and observing their movement.
- Changing materials or gravity conditions.
- Visualising velocities, contact points and impact forces.
Object positions and scene changes are stored in the intermediate representation and its associated programs. Users can therefore follow how an environment evolves after an interaction, rather than only watching a video sequence.
Training robots in simulation for navigation and object manipulation
One of Mundo’s applications is preparing robotic tasks. The model can build customised scenes in which different robots can be represented during navigation or manipulation operations.
For navigation, the 3D representation provides object categories, their positions, obstacles and traversable areas. A planner uses this information to search for paths. Mundo can then generate first-person or third-person views showing the robots’ virtual movements.
For manipulation, the system links the geometry of objects to their physical properties. The tasks presented include placing an apple in a bowl, setting a can on a tray and stacking blocks.
These exercises can be represented with robots of different morphologies. Mundo generates external views as well as images corresponding to cameras mounted on the robots’ heads or wrists.
The system also stores task trajectories and the states of physical interactions. This correspondence between visual observations and computed movements makes it possible to examine the steps of a manipulation and compare different strategies in a simulated environment.
Space-time exploration: replaying robot actions from any angle
Mundo also offers a time exploration feature. Dynamic events captured by several cameras can be rendered from new viewpoints.
In scenes simulated from images, movements are based on gravity and contact dynamics calculations. Users can pause a sequence at a given moment, move the virtual camera and watch the same action from different angles.
This feature, comparable to a bullet-time effect, is used to examine trajectories, contacts and occlusion relationships between objects during an action.
How Mundo works with Kairos to prepare robots for real environments
Mundo is part of a world-model architecture with Kairos at its core. It adds complementary functions: reconstructing specific environments, generating training data and physics simulation.
The model is also paired with Astra to develop spatial understanding capabilities. This combination aims to connect the representation of environments, their programmable modification and the analysis of robot interactions.
The intended industrial application is to use data collected on site to build virtual scenes matching the robots’ future workplaces. Trajectories and manipulation strategies can then be studied in these environments before they are implemented physically.
The capabilities presented concern the reconstruction and simulation of environments and the generation of robot observations. The examples described are virtual demonstrations, not measured performance of physical robots in working conditions.