[12/12]
We believe ESWM points to a new generation of brain-inspired models—ones that reason over fragments, generalize across structure, and adapt on the fly.

📄 arxiv.org/abs/2505.13696
👥 @maximemdaigle.bsky.social , @bashivan.bsky.social

[11/12]
🔧 When environments change—say a new wall appears—ESWM adapts instantly. No retraining is needed. Just update the memory bank and the model replans.

This separation of memory and reasoning makes ESWM highly flexible.

[10/12]
🧭 It gets even better!

ESWM can navigate between arbitrary points using only its memory bank—planning efficiently in latent space with near-optimal paths.

No access to global maps or coordinates required.

[9/12]
🚶 With no additional training, ESWM can explore novel environments efficiently by acting on uncertainty.

[8/12]
⚙️ How are these maps built?

We find that ESWM stitches together memories via overlapping states—merging local transitions into global structure.

Obstacles and boundaries serve as spatial anchors, guiding how memories are organized in latent space.

[7/12]
🏞️ How does ESWM solve the task?

Using ISOMAP, we visualize its latent representations—beautifully organized spatial layouts emerge from its internal states, even when the model sees only a small part or out-of-distribution environments.

[6/12]
⚡️ Transformer-based ESWM models outperform LSTMs and Mamba, especially in settings where observations are compositional. Attention allows the model to flexibly bind relevant memories and generalize across structures.

[5/12]
To train ESWM, we use meta-learning across diverse environments. At test time, the model gets a minimal set of disjoint episodic memories (single transitions) and must predict a missing element in a new transition—without ever seeing the full map.

[4/12]
🧠 Inspired by the MTL’s architecture and function, we built ESWM: a neural network that infers the structure of its environment from isolated, one-step transitions—just like the brain integrates episodes into a cognitive map.

[3/12]
Such shortcut-seeking behavior is supported by internal world models in the brain’s medial temporal lobe (MTL), an area involved in both episodic memory and spatial navigation.

[2/12]
Many animals can infer unexperienced paths by integrating disjoint memories. Mice, e.g., take shortcuts they’ve never physically traversed: pubmed.ncbi.nlm.nih.gov/17469957/

🧠 Can a neural network build a spatial map from scattered episodic experiences like humans do?

We introduce the Episodic Spatial World Model (ESWM)—a model that constructs flexible internal world models from sparse, disjoint memories.

🧵👇 [1/12]