A huge thanks to my fantastic mentors, collaborators and advisors: Andrew Bolton, Vikash Mansinghka, @joshtenenbaum.bsky.social and @thiskevinsmith.bsky.social. Do check out our project page (arijit-dasgupta.github.io/jtap/) for more details and the code to run this experiment.

This can be explained with an intuitive example. If the object does not come out the left side, it makes sense to think it might have bounced and will eventually hit red. Just like humans, JTAP can predict this, showing how the lack of changing visual evidence is evidence itself.

To test whether humans update their beliefs during occlusion, we compared against two ablative baselines that either decay or freeze beliefs when the object is hidden. A targeted analysis of occluded time-steps shows that the full JTAP model captures human behavior much better.

And when we do this, we find that we do really well at predicting overall time-aligned human behavior, both on when to make a decision, and which decision to make. Not only that, we also capture those moments of graded uncertainty, when humans don’t have full agreement.

A single JTAP run acts as a proxy for one pseudo-participant’s belief states over time. By feeding multiple randomly-seeded runs into a decision model accounting for time delay and hysteresis, we predict both individual-level human decisions and aggregate behavior.

JTAP produces current and future beliefs over a single object’s position, speed, & direction. In this example, it predicts that future beliefs over red vs green outcomes spread under occlusion, then sharpen to red once visible, while in the visible case they stay confidently red.

We approach this with the Joint Tracking and Prediction (JTAP) model: a structured probabilistic program that combines 2D physical dynamics with an observation model. Posterior beliefs are inferred via a particle-based approximation using a Sequential Monte Carlo algorithm.

Computationally, we frame this as a Bayesian inference problem: jointly inferring an observer’s beliefs about an object’s current and future state from a stream of images.

We started by asking participants to continuously predict whether a ball moving in a 2.5D scene with gray occluders and black barriers would hit the red or green region first. People showed varying uncertainty in their responses, while giving reliable responses (ICC1k = 0.952).

Although often studied separately, we explore whether people jointly integrate state uncertainty from occlusion and uncertainty about future physical dynamics to form intuitive beliefs about what will happen next. Can machines capture this kind of physically grounded reasoning?