Predictive Spatio-Temporal Scene Graphs for Semi-Static Scenes

IEEE Robotics and Automation Letters (RA-L) 2026
Miguel Saavedra-Ruiz*,1, Charlie Gauthier*,1, Kumaraditya Gupta1, Shima Shahfar1, Kirsty Ellis1, Steven Parkison2, and Liam Paull1,3
1 DIRO, Université de Montréal and Mila 2 Hydro-Québec Research Institute 3 Canada CIFAR AI Chair
* Equal contribution
Paper arXiv Codeout soon! Data

Abstract

We have seen tremendous recent progress in our ability to build "spatio-semantic" representations that enable robots to perform complex reasoning across geometry and semantics. However, the vast majority of these methods lack any ability to perform reasoning across time. This is a desirable property in situations where a robot repeatedly observes an environment where instances may change in between observations, but in a structured way. Consider as an example a home environment where the location of a mug typically moves from the cupboard to a countertop to the sink and then back to the cupboard on a daily basis. We should be able to learn this cyclic behavior and use it to predict the state of the mug in the future. In this work, we propose a method that is able to perform this type of tempo-spatio-semantic reasoning. Underpinning the method is a filter, Perpetua*, that performs Bayesian reasoning on the states of the environment that are observed over time. This filter is integrated within a 3D scene graph structure that we call PredictiveGraphs, where nodes represent objects and edges function as Perpetua* filters encoding spatio-semantic relationships. We validate the method in both simulation and real-world dynamic navigation tasks, where our real-world experiments consist of an environment that is undergoing semi-static changes at a bi-hourly frequency over a period of three weeks. In both settings, we demonstrate that our method outperforms baselines in predicting future environment states, even in the presence of distributional shifts.

Video

How it works

An alarm clock node connected to three receptacles (bed, chair, TV stand) through Perpetua* edges with probabilities 0.85, 0.13 and 0.02.
Edge Update: Each semi-static object node has an associated set of Perpetua* estimators (one for each receptacle node it has been observed at). At prediction time, Perpetua* updates the belief of each edge, indicating the object's most likely receptacle.
Log prior, likelihood and posterior odds over time for Perpetua*'s emergence and persistence models.
Perpetua* Tempered Bayesian Model Averaging Positive values favor the emergence model (𝓜E), which captures object appearance, while negative values favor the persistence model (𝓜P), which captures disappearance. The prior encodes long-term patterns, while the likelihood captures recent observations, letting the model adapt as new data arrives. When observations stop (ta to tb), the likelihood fades and the posterior is dominated by the prior, allowing the model to rely on previously learned patterns.

Simulation Experiments

"I need to send an email, but I can't find my laptop."
PredictiveGraphs predicts the laptop has moved from where it was last seen and navigates straight to its new location.
A semi-static home
2×
Objects move, vanish and reappear over time.
Finding the alarm clock
20×
Predicted on the chair.
Finding the soap bottle
10×
Predicted on the side table.

Results

Bar plots of success and SPL for static, dynamic and absent objects comparing CG, CG+LLM, CG+LLM+History, HOMER, FlowMaps and PredictiveGraphs, under privileged and CLIP+LLM perception.
Main prediction benchmark. PredictiveGraphs with Perpetua* or FreMEn estimators outperforms baselines under both privileged and noisy perception.
All 15 simulated homes
Top-down views of the 15 ProcTHOR environments used in the experiments.

ProcTHOR environments: Top-down view of the different 15 ProcTHOR scenes used in the simulation experiments. Each scene has at least 5 rooms and a navigable space between 100 m² and 150 m².

Real-World Experiments

Top-down and perspective point clouds of the three-room laboratory, with receptacles outlined.
  • 3 weeks of semi-static changes, updated every 2 hours
  • 3 rooms, 28 receptacles, 21 objects
  • 100 open-vocabulary queries, up to a week ahead
  • Agilex Ranger Mini 2.0 with a RealSense D435 camera
"Is there something useful to comb my hair?"
The graph predicts the hair brush is in another room and that the door on the shortest path is closed, so the robot takes the longer, open route.
"Could you get me a healthy vegetable?"
The carrot isn't at the most likely receptacle, so the robot moves on to the next-best prediction and finds it there.

What the robot sees on arrival

Segmented camera view showing a hair brush on a table.
Hair brush
Segmented camera view showing a carrot on a desk.
Carrot
Segmented camera view showing a panda plushie on a table.
Panda plushie

Results

Predictive navigation door experiment showing blocked shortest path anticipation and feasible replanning.
Predictive Navigation: PredictiveGraphs uses its predictive capabilities to anticipate that the shortest path to its goal is blocked (low probability). This allows it to preemptively adapt its navigation plan, choosing a path that is longer but feasible (high probability), and successfully reaching the target without visiting unnecessary locations.
Bar plots for adaptive navigation in simulation (left) and real-world navigation (right) comparing CG, DualMap, HOMER, FlowMaps and PredictiveGraphs.
Adaptation and real-world navigation. Perpetua* improves performance under distributional shifts, while PredictiveGraphs supports robust dynamic navigation with noisy real-world data.

Citation

@article{saavedra2026predictive,
  title   = {Predictive Spatio-Temporal Scene Graphs for Semi-Static Scenes},
  author  = {Saavedra-Ruiz, Miguel and Gauthier, Charlie and Gupta, Kumaraditya and
             Shahfar, Shima and Ellis, Kirsty and Parkison, Steven and Paull, Liam},
  journal = {IEEE Robotics and Automation Letters},
  year    = {2026}
}