Machine learning researchers have developed a technique that allows world models to make better predictions about how objects will behave. World models are used to train physical AI systems like robots how to navigate the real world. They predict the consequences of actions. But they do so based on observed data. These models could provide better direction to robots if they were able to make inferences about possible actions without being trained for all the possible outcomes. Randall Balestriero (Brown University, AMI Labs), Kerda Varaku, and Turing Award winner Yann LeCun (NYU, AMI Labs) describe an approach that might allow that to happen in a pre-print paper, "Counterfactual World Models: Natura Non Facit Saltus." The Latin refers to Gottfried Leibniz's axiom "nature makes no leaps," which had a good run until it ran headlong into quantum mechanics. Yet for the authors, the saying explains how they have been able to devise a method for zero-shot physics inference. Counterfactual world models, said Balestriero in an email to The Register use "the mismatch between your prediction and the real world to figure out what did you get wrong, and from that you can better understand what happened for better future predictions, debugging, reasoning, all at test time." Joint Embedding Predictive Architecture (JEPA) models, developed by LeCunn and colleagues back in 2023, have been proposed for robots because they can take an action as input and ask what the world would be like with that action. But a JEPA world model produces an average of outcomes and would not be able to anticipate counterfactual scenarios – events that didn't happen. The technique that the researchers developed is not specific to JEPAs – it can be applied to any world model that predicts the average next state under any action. But JEPA models are particularly well suited to being adapted. Balestriero said that the counterfactual world model experiments were done using off-the-shelf encoders (DINOv2 and MAE) without any fine-tuning and only a small predictor trained on top. Creating a counterfactual world model from that starting point would only require minor additions. Counterfactual world models predict alternative scenarios more effectively by inferring possibilities from what did occur. "In causal inference, answering 'What would have happened had I acted differently?' takes three steps," explained Balestriero. "You use what you observed to learn something about those hidden factors, you change the action, and you predict again with the hidden factors kept exactly as they were. "Our counterfactual world model does all three steps with one subtraction." Balestriero cited an example from the paper: a boat crossing a river with a hidden current and how the JEPA world model and counterfactual world model predicted the boat's movements. "The world model predicted where the boat would end up after steering left, and it actually ended up two meters further downstream," he explained. "Those two meters are the signature of the current that occurred. To answer 'where would the boat be had I steered right?', we take the world model's prediction for steering right and add the same two meters. In other words, the counterfactual is what actually happened, plus the difference between the model's predictions for the alternative action and for the action taken. "Physics enters to justify this rule, not as an equation the model fits. Over a short time, the effect of the action and the effect of the current each grow in proportion to the duration, while their interaction grows only with its square, so over short steps they simply add up, like forces. "We prove that the fraction of the disturbance's effect that this rule gets wrong shrinks with the square of the time step, and that no method using the same ordinary data can do better than this rate in the worst case." Counterfactual world models have three likely uses, according to Balestriero. The first is having a robot retry an action after a surprise event. "When an attempt fails because of something the robot could not see, such as an unexpected load, a current or a weak motor, the counterfactual world model lets it estimate which action would have worked in that same situation," he explained. "In our simulated tasks, choosing the action this way reaches the goal in 35 to 69 percent of events, against 14 to 23 percent for a planner that knows the true simulator but ignores the observed outcome. With two more attempts on the same event, success rises to 51 to 97 percent." He also sees these models being useful for adapting to persistent fault, like a miscalibrated motor, and helping people understand failures. "World models are trained to predict, and they are getting good at it," said Balestriero. "But a robot, a car or a drone acting in the real world is constantly surprised by things it could not see coming. After a surprise, the useful question is no longer what was expected, but what would have happened had it acted differently in that same situation. You need that answer to debug a failure, to tell a bad decision from bad luck, or to decide what to do on the next attempt, and prediction alone does not provide it." The researchers initial attempts to address the issue involved adding a learned component to the world model that infers whatever hidden disturbance (e.g. the current in the boat scenario) affected the anticipated outcome. But that approach was too fragile. "Trying to understand why led to the theory," explained Balestriero. "Learning the disturbance from ordinary data is not just hard, it is impossible in general: the hidden noise can be relabeled differently for each action without changing a single prediction, so the data cannot tell you which labeling is the right one. The missing piece has to come from outside the data. For physical systems, it comes from the fact that forces add and that their interaction builds up slowly in time. Once you accept that, the learned disturbance model becomes unnecessary: the gap between what happened and what was predicted already carries the information you need, and you carry it over to the other action." Balestriero said that control engineers have been implementing corrections of this sort for decades, to tune hand-built models. "What was missing was why the same correction is justified for learned world models that see the world through images, what can and cannot be recovered without paired data, and where it breaks," he said. "The paper answers these questions by connecting causal inference, control and world models. The title comes from Leibniz's law of continuity, natura non facit saltus, 'nature makes no leaps,' which is precisely the property that makes all of this work." The researchers' paper was provided to The Register prior to its release on ArXiv. It is expected to be released this week. ®
/s3/static.nrc.nl/wp-content/uploads/2026/11/09170241/Untitled-Design-3.jpg)
/s3/static.nrc.nl/wp-content/uploads/2026/11/09165115/web-091126ECO_2038376785_jensen.jpg)
