Our work showing human-like individual differences in perceptual decisions emerge from random weight initializations in deep neural networks has been accepted in two NeurIPS workshops! 🎉 Awesome job by my student @herrickfung.bsky.social in collaboration with the amazing @apurvaratan.bsky.social.

Excited to share new work with @hleemasson.bsky.social , Ericka Wodka, Stewart Mostofsky and @lisik.bsky.social! We investigated how simultaneous vision and language signals are combined in the brain using naturalistic+controlled fMRI. Read the paper here: osf.io/b5p4n
1/n

Check out this new preprint led by @gkathy.bsky.social showing how a new fine-tuning scheme using human similarity judgements can improve existing video models, making them more human-aligned on different social tasks!

arxiv.org/abs/2510.01502

Together this work shows how different types of human similarity judgments can be leveraged to improve video models.

We also share our large-scale video similarity judgment dataset, and code for hybrid triplet/RSA behavior guided fine-tuning: github.com/garciakathy/...

In follow-up experiments we show this model generalizes better to novel social tasks, and avoids catastrophic forgetting by preserving baseline on action recognition tasks.

After fine-tuning, the video model explains both captures more shared variance with language models AND captures more unique variance in human judgments, indicating it learned both language-like semantics and additional visual social nuances.

Hybrid fine-tuning substantially increases match to human judgment on held-out videos and surpasses the best language model baseline. The hybrid loss > triplet-only loss and > RSA-only loss.

We fine-tune a video transformer with a novel hybrid objective function = Triplet loss (local constraints) + RSA Loss (global Pearson-correlation over pairwise distances), which captures both local and global similarity structure. We use Low Rank Adaptation to reduce the number of trainable params.

Despite the task being purely visual, caption embeddings from a language model predict human similarity better than any pretrained video model (e.g., mpnet-base-v2 > TimeSformer-base).

🚨New preprint w/ @lisik.bsky.social!
Aligning Video Models with Human Social Judgments via Behavior-Guided Fine-Tuning

We introduce a ~49k triplet social video dataset, uncover a modality gap (language > video), and close via novel behavior-guided fine-tuning.
🔗 arxiv.org/abs/2510.01502

Excited to be presenting this paper at #ICLR2025 this week!
Come to the poster if you want to know more about how human brains and DNNs process video 🧠🤖

📆 Sat 26 Apr, 10:00-12:30 - Poster session 5 (#64)
📄 openreview.net/pdf?id=LM4PY...
🌐 sergeantchris.github.io/hundred_mode...

🚀 Together, this work highlights a major gap in AI's ability to match human social vision, and underscores the importance of developing AI models in dynamic social contexts [6/6]

📹 While most model features (like architecture or training objective) did not affect performance, we saw a big advantage for video versus image models along the lateral stream. But no model tested could predict anterior lateral stream responses well. [5/6]

🔍 Unlike visual scene features and ventral stream responses, vision models struggled to match human action and social interaction ratings, and did a poor job of predicting brain responses along the recently proposed lateral stream, specialized for social perception. [4/6]

🧠 We benchmarked 350+ image, video, and language models against human behavioral and neural responses to dynamic, social videos. [3/6]

🎥 Real-world vision is dynamic, involving complex social interactions. Current AI models provide a good match to humans in static scene vision, but how do they fare with dynamic, social stimuli? 🤔 We set out to explore this! [2/6]

📢 Excited to announce our paper at #ICLR2025: “Modeling dynamic social vision highlights gaps between deep learning and humans”! w/ @emaliemcmahon.bsky.social, Colin Conwell, Mick Bonner, @lisik.bsky.social
‬

‪📆 Thur, Apr, 24: 3:00-5:30 - Poster session 2 (#64) ‬
‪📄 bit.ly/4jISKES%E2%8... [1/6]

New paper! 🧠 **The cerebellar components of the human language network**

with: @hsmall.bsky.social @moshepoliak.bsky.social @gretatuckute.bsky.social @benlipkin.bsky.social @awolna.bsky.social @aniladmello.bsky.social and @evfedorenko.bsky.social

www.biorxiv.org/content/10.1...

1/n 🧵

Congratulations to @emaliemcmahon.bsky.social and all the Glushko Prize winners on this amazing and well-deserved honor!

It is truly an honor to be recognized as a Glushko prize winner among all these incredible scientists. Thank you to the prize committee and to my PhD advisors @lisik.bsky.social and Mick Bonner.