Lightnews — Scholar-powered news

Reposted by T. Anderson Keller

Kempner Institute at Harvard University @kempnerinstitute.bsky.social · Jul 22

New in the #DeeperLearningBlog: #KempnerInstitute research fellow @andykeller.bsky.social introduces the first flow equivariant neural networks, which reflect motion symmetries, greatly enhancing generalization and sequence modeling.

bit.ly/451fQ48

#AI #NeuroAI

Flow Equivariant Recurrent Neural Networks - Kempner Institute

Sequence transformations, like visual motion, dominate the world around us, but are poorly handled by current models. We introduce the first flow equivariant models that respect these motion symmetrie...

bit.ly

4 8

Reposted by T. Anderson Keller

Kanaka Rajan @kanakarajanphd.bsky.social · Jul 2

(1/7) New preprint from Rajan lab! 🧠🤖
@ryanpaulbadman1.bsky.social & Riley Simmons-Edler show–through cog sci, neuro & ethology–how an AI agent with fewer ‘neurons’ than an insect can forage, find safety & dodge predators in a virtual world. Here's what we built

Preprint: arxiv.org/pdf/2506.06981

3 32 94

Reposted by T. Anderson Keller

Nick Blauch @nblauch.bsky.social · Jun 16

What shapes the topography of high-level visual cortex?

Excited to share a new pre-print addressing this question with connectivity-constrained interactive topographic networks, titled "Retinotopic scaffolding of high-level vision", w/ Marlene Behrmann & David Plaut.

🧵 ↓ 1/n

1 24 67

Reposted by T. Anderson Keller

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · May 23

Are you an RL PhD at Harvard who has had your funding wrecked by the government and working on topics related to multi-agent? Reach out, I am happy to try to find a way to support you.

1 7 50

Reposted by T. Anderson Keller

DrBreaky @drbreaky.bsky.social · May 6

Looking forward to presenting our work on cortico-hippocampal coupling and wave-wave interactions as a basis for some core human cognitions

5pm May 6th EST (US)
8am May 7th AEST (Sydney)

Zoom link: columbiacuimc.zoom.us/j/92736430185

Thanks to WaveClub conveners Erfan Zabeh & Uma Mohan

2 4 19

Reposted by T. Anderson Keller

Kempner Institute at Harvard University @kempnerinstitute.bsky.social · Mar 29

It’s another big day for the #KempnerInstitute at @CosyneMeeting! Check out our work highlighted in poster session 3 today! #COSYNE2025

1 2 12

T. Anderson Keller @andykeller.bsky.social · Mar 12

Such a cool connection!! I never heard of that, but that is an ingenious solution. I will likely use this reference in my future talks and mention your comment if you don’t mind!

1 1

T. Anderson Keller @andykeller.bsky.social · Mar 11

Thanks for reading! Can you explain your thought process here? Imagine a neuron with a receptive field (size of the yellow square) localized to the center of the pentagon. Its input would be entirely white — same as if it were localized to the center of the triangle; and therefore indistinguishable.

1

Reposted by T. Anderson Keller

Kevin Mitchell @wiringthebrain.bsky.social · Mar 10

Super interesting thread!

T. Anderson Keller @andykeller.bsky.social · Mar 10

In the physical world, almost all information is transmitted through traveling waves -- why should it be any different in your neural network?

Super excited to share recent work with the brilliant @mozesjacobs.bsky.social: "Traveling Waves Integrate Spatial Information Through Time"

1/14

2 2 38

T. Anderson Keller @andykeller.bsky.social · Mar 10

And not to forget, a huge thanks to all those involved in the work: Lyle Muller, Roberto Budzinski & Demba Ba!! And further thanks to those who advised me and shaped my thoughts on these ideas @wellingmax.bsky.social & Terry Sejnowski. This work would not have been possible without their guidance.

3

Reposted by T. Anderson Keller

Mozes Jacobs @mozesjacobs.bsky.social · Mar 10

Traveling waves of neural activity are observed all over the brain. Can they be used to augment neural networks?

I am thrilled to share our new work, "Traveling Waves Integrate Spatial Information Through Time" with @andykeller.bsky.social!

1/13

2 10 38

Reposted by T. Anderson Keller

Yohan J John @dryohanjohn.bsky.social · Mar 10

Really interesting RNN work.

And based on some spiking simulations I've tinkered with, it seems plausible that PV, CB & CR interneurons can contribute to changing the boundary conditions and the 'elasticity' of the oscillating 'rubber sheet' of cortex (and probably hippocampus and amygdala too). 🤓

T. Anderson Keller @andykeller.bsky.social · Mar 10

In the physical world, almost all information is transmitted through traveling waves -- why should it be any different in your neural network?

Super excited to share recent work with the brilliant @mozesjacobs.bsky.social: "Traveling Waves Integrate Spatial Information Through Time"

1/14

2 10

T. Anderson Keller @andykeller.bsky.social · Mar 10

For all the technical details and more ablations, please see our paper recently accepted in workshop-form at ICLR Re-Align, and full-version preprint on ArXiv!

Paper: arxiv.org/abs/2502.06034
Code: github.com/KempnerInsti...

Hope to see you in Singapore!

Fin/

Traveling Waves Integrate Spatial Information Through Time

Traveling waves of neural activity are widely observed in the brain, but their precise computational function remains unclear. One prominent hypothesis is that they enable the transfer and integration...

arxiv.org

1 9

T. Anderson Keller @andykeller.bsky.social · Mar 10

If you want more visualizations, a bit more depth, and even some audio of what different images 'sound' like to our models, please check out our @kempnerinstitute.bsky.social blog-post!

kempnerinstitute.harvard.edu/research/dee...

13/14

Traveling Waves Integrate Spatial Information Through Time - Kempner Institute

The act of vision is a coordinated activity involving millions of neurons in the visual cortex, which communicate over distances spanning up to centimeters on the cortical surface. How do […]

kempnerinstitute.harvard.edu

1 1 10

T. Anderson Keller @andykeller.bsky.social · Mar 10

Overall, we believe this is the first step of many towards creating neural networks with alternative methods of information integration, beyond those that we have currently such as network depth, bottlenecks, or all-to-all connectivity, like in Transformer self-attention.

12/14

1 5

T. Anderson Keller @andykeller.bsky.social · Mar 10

We found that wave-based models converged much more reliably than deep CNNs, and even outperformed U-Nets with similar numbers parameter when pushed to their limits. We hypothesize that this is due to the parallel processing ability that wave-dynamics confer and other CNNs lack.

11/14

Tables from the paper comparing wave based models and baselines (CNNs and U-Nets) on a variety of semantic segmentation tasks

1 6

T. Anderson Keller @andykeller.bsky.social · Mar 10

As a first step towards the answer, we used the Tetris-like dataset and variants of MNIST to compare the semantic segmentation ability of these wave-based models (seen below) with two relevant baselines: Deep CNNs w/ large (full-image) receptive fields, and small U-Nets.

10/14

1 1 7

T. Anderson Keller @andykeller.bsky.social · Mar 10

We were super excited about these results—they aligned with the long-standing hypothesis that traveling waves integrate spatial information in the brain*. But does this hold any practical implications for modern machine learning?

pubmed.ncbi.nlm.nih.gov/7947408
www.science.org/doi/abs/10.1...

9/14

Horizontal Propagation of Visual Activity in the Synaptic Integration Field of Area 17 Neurons

The receptive field of a visual neuron is classically defined as the region of space (or retina) where a visual stimulus evokes a change in its firing activity. At the cortical level, a challenging is...

www.science.org

1 5

T. Anderson Keller @andykeller.bsky.social · Mar 10

Was this just due to using Fourier transforms for semantic readouts, or wave-biased architectures? No! The same models with LSTM dynamics and a linear readout of the hidden-state timeseries still learned waves when trying to semantically segment images of Tetris-like blocks!

8/14

1 5

T. Anderson Keller @andykeller.bsky.social · Mar 10

Looking at the Fourier transform of the resulting neural oscillations at each point in the hidden state, we then saw that the model learned to produce different frequency spectra for each shape, meaning each neuron really was able to 'hear' which shape it was a part of!

7/14

Plot of five representative frequency bins from the FFT of the dynamics of our wave-RNN on the shape task. We see different shapes pop out in different bins, indicating that they 'sound' different, and allowing the model to uniquely classify each shape. On the right we plot the average FFT for each pixel, separated by each shape, over the whole dataset, showing that different shapes do have measurably different frequency spectra, even in this average case.

1 7

T. Anderson Keller @andykeller.bsky.social · Mar 10

We made wave dynamics flexible by adding learned damping and natural frequency encoders, allowing hidden state dynamics to adapt based on the input stimulus. On simple polygon images, we found the model learned to use these parameters to produce shape-specific wave dynamics:

6/14

2 7

T. Anderson Keller @andykeller.bsky.social · Mar 10

To test this, we needed a task; so we opted for semantic segmentation on large images, but crucially with neurons having very small one-step receptive fields. Thus, if we were able to decode global shape information from each neuron, it must be coming from recurrent dynamics.

5/14

Visualization of the input stimuli to our network (left) and the target segmentation labels by color (right). The receptive field of the final layer neurons in our model is plotted as the yellow box, demonstrating that a single neuron has no way to know what shape it may be a part of simply from its local neighborhood, and therefore will require global integration of information over time to solve the task.

2 6

T. Anderson Keller @andykeller.bsky.social · Mar 10

We found that, in-line with theory, we could reliably predict the area of the drum analytically by looking at the fundamental frequency of oscillations of each neuron in our hidden state. But is this too simple? How much further can we take it if we add learnable parameters?

4/14

Visualization of the same wave-based RNN on two drums of different sizes (13 and 33 side length respectively). In the middle (in purple) we show the displacement of the drum head at a point just off the center, and (in red) the theoretical fundamental frequency of vibration that we can analytically derive for a square of side length L plotted. On the right we show the Fourier transform of these time-series dynamics, showing the frequency peak in the expected location. This validates we can estimate the size of a drum head from the frequency spectrum of vibration at any point.

1 1 6

T. Anderson Keller @andykeller.bsky.social · Mar 10

Inspired by Mark Kac’s famous question, "Can one hear the shape of a drum?" we thought: Maybe a neural network can use wave dynamics to integrate spatial information and effectively "hear" visual shapes... To test this, we tried feeding images of squares to a wave-based RNN:

3/14

1 1 9

T. Anderson Keller @andykeller.bsky.social · Mar 10

Just as ripples in water carry information across a pond, traveling waves of activity in the brain have long been hypothesized to carry information from one region of cortex to another (Sato 2012)*; but how can a neural network actually leverage this information?

* www.cell.com/neuron/fullt...
2/14

Traveling Waves in Visual Cortex

In this Review, Sato et al. summarize the evidence in favor of traveling waves in primary visual cortex. The authors suggest that their substrate may lie in long-range horizontal connections and that ...

www.cell.com

1 1 7