Lightnews — Scholar-powered news

Guy Davidson @guydav.bsky.social · 19d

Belated update #2: my year at Meta FAIR through the AIM program was so nice that I’m sticking around for the long haul.

I’m excited to stay at FAIR and work with @asli-celikyilmaz.bsky.social and friends on fun LLM questions; I’ll be working from the New York office so we’re sticking around.

1 7

Guy Davidson @guydav.bsky.social · 21d

Thank you, Ed!!

Guy Davidson @guydav.bsky.social · 21d

Tune in tomorrow for belated update #2, on post-PhD plans!

Guy Davidson @guydav.bsky.social · 21d

I owe tremendous thanks to many other people, all (or, hopefully, at least most) of whom I mentioned in my acknowledgments. I’m also so grateful my dad could represent my family, and for my wife, Sarah, for, well, everything.

1

Guy Davidson @guydav.bsky.social · 21d

Much, much larger thanks to my advisors, @brendenlake.bsky.social and @toddgureckis.bsky.social , for your guidance and mentorship over the last several years. I appreciate you so much, and this wouldn’t have looked the same without you!

1 2

Guy Davidson @guydav.bsky.social · 21d

Belated update #1: I defended my PhD about a month ago! I appreciate the warm reception from everyone who made it in-person and virtually. Thanks to my committee, @lerrelpinto.com, @togelius.bsky.social, and @markkho.bsky.social for your feedback and fun questions.

3 30

Guy Davidson @guydav.bsky.social · Aug 6

Friends and virtual acquaintances! I’m defending my PhD tomorrow morning at 11:30 AM ET. If anyone would like to watch, let me know and I’ll send you the Zoom link (and if you’re in NYC and feel compelled to join in person, that works, too!)

8

Guy Davidson @guydav.bsky.social · Jul 30

Wherever good coffee is to be found, the rest of the time. Don't hesitate to reach out!

(also happy to talk about job search in industry and what that looks and feels like these days)

Guy Davidson @guydav.bsky.social · Jul 30

Saturday's poster session (P3-D-44) to talk about our goal inference work, in a new, physics-based environment we developed: escholarship.org/uc/item/6tb2...

Goal Inference using Reward-Producing Programs in a Novel Physics Environment

Author(s): Davidson, Guy; Todd, Graham; Colas, CŽdric; Chu, Junyi; Togelius, Julian; Tenenbaum, Joshua B.; Gureckis, Todd M; Lake, Brenden | Abstract: A child invents a game, describes its rules, and ...

escholarship.org

1

Guy Davidson @guydav.bsky.social · Jul 30

Today's Minds in the Making: Design Thinking and Cognitive Science Workshop (Pacific E):

minds-making.github.io

1 1

Guy Davidson @guydav.bsky.social · Jul 30

#CogSci2025 friends! I'm here all week and would love to chat. I'd particularly love to talk to anyone thinking about Theory of Mind and how to evaluate it better (in both minds and machines, in different settings and contexts), and about goals and their representations. Find me at:

1 1 7

Guy Davidson @guydav.bsky.social · Jul 8

Cool new work on localizing and removing concepts using attention heads from colleagues at NYU and Meta!

Karen Ullrich (s/h) ✈️ COLM @karen-ullrich.bsky.social · Jul 8

How would you make an LLM "forget" the concept of dog — or any other arbitrary concept? 🐶❓

We introduce SAMD & SAMI — a novel, concept-agnostic approach to identify and manipulate attention modules in transformers.

3

Guy Davidson @guydav.bsky.social · Jun 6

You (yes, you!) should work with Sydney! Either short-term this summer, or longer term at her nascent lab at NYU!

Sydney Levine @sydneylevine.bsky.social · Jun 6

🔆 I'm hiring! 🔆

There are two open positions:

1. Summer research position (best for master's or graduate student); focus on computational social cognition.
2. Postdoc (currently interviewing!); focus on computational social cognition and AI safety.

sites.google.com/corp/site/sy...

Sydney Levine - Open Positions

Summer Research Position I am seeking a part-time or full-time researcher for the summer (starting asap) to bring a project to completion. The project asks the question: do people around the world u...

sites.google.com

4

Guy Davidson @guydav.bsky.social · May 30

Fantastic new work by @johnchen6.bsky.social (with @brendenlake.bsky.social and me trying not to cause too much trouble).

We study systematic generalization in a safety setting and find LLMs struggle to consistently respond safely when we vary how we ask naive questions. More analyses in the paper!

John (Yueh-Han) Chen @johnchen6.bsky.social · May 29

Do LLMs show systematic generalization of safety facts to novel scenarios?

Introducing our work SAGE-Eval, a benchmark consisting of 100+ safety facts and 10k+ scenarios to test this!

- Claude-3.7-Sonnet passes only 57% of facts evaluated
- o1 and o3-mini passed <45%! 🧵

3 10

Guy Davidson @guydav.bsky.social · May 23

Finally, if this work makes you think "I'd like to work with this person," please reach out -- I'm on the job market for industry post-PhD roles (keywords: language models, interpretability, open-endedness, user intent understanding, alignment).
See more: guydavidson.me

Guy Davidson

Guy Davidson's academic website

guydavidson.me

3

Guy Davidson @guydav.bsky.social · May 23

If you made it this far, thank you, and don't hesitate to reach out! 17/N=17
Paper: arxiv.org/abs/2505.12075
Code: github.com/guydav/promp...

Do different prompting methods yield a common task representation in language models?

Demonstrations and instructions are two primary approaches for prompting language models to perform in-context learning (ICL) tasks. Do identical tasks elicited in different ways result in similar rep...

arxiv.org

1 2

Guy Davidson @guydav.bsky.social · May 23

As with pretty much everything else I've worked on in grad school, this work would have looked different (and almost certainly worse) without the guidance of my advisors, @brendenlake.bsky.social and @toddgureckis.bsky.social . I continue to appreciate your thoughtful engagement with my work! 16/N

1

Guy Davidson @guydav.bsky.social · May 23

This work would also have been impossible without @adinawilliams.bsky.social 's guidance, the freedom she gave me in picking a problem to study, and believing in me that I could tackle it despite it being my first foray into (mechanistic) interpretability work. 15/N

1

Guy Davidson @guydav.bsky.social · May 23

We owe a great deal of gratitude to @ericwtodd.bsky.social d , not only for open-sourcing their code, but also for answering our numerous questions over the last few months. If you find this interesting, you should also read their paper introducing function vectors. 14/N

1

Guy Davidson @guydav.bsky.social · May 23

See the paper for a description of the methods, the many different controls we ran, our discussion and limitations, examples of our instructions and baselines, and other odd findings (applying an FV twice can be beneficial! Some attention heads have negative causal effects!) 13/N

1

Guy Davidson @guydav.bsky.social · May 23

Finding 5 bonus: Which post-training steps facilitate this? Using the OLMo-2 model family, we find that the SFT and DPO stages each bring a jump in performance, but the final RLVR step doesn't make a difference for the ability to extract instruction FVs. 12/N

1

Guy Davidson @guydav.bsky.social · May 23

Finding 5: We can steer base models with instruction FVs extracted from their post-trained versions. We didn't expect this to work! It's less effective for the Llama-3.2 models that are distilled and smaller. We're also excited to dig into this and see where we can push it. 11/N

1

Guy Davidson @guydav.bsky.social · May 23

Finding 4: The relationship between demonstrations and instructions is asymmetrical. Especially in post-trained models, the top attention heads for instructions appear peripherally useful for demonstrations, more than the opposite case (see paper for details). 10/N

1

Guy Davidson @guydav.bsky.social · May 23

We (preliminarily) interpret this as evidence that the effect of post-training is _not_ in adapting the model to represent instructions with the mechanism used for demonstrations, but in developing a mostly complementary mechanism. We're excited to dig into this further. 9/N.

1

Guy Davidson @guydav.bsky.social · May 23

Finding 3 bonus: examining activations in the shared attention heads, we see (a) generally increased similarity with increasing model depth, and (b) no difference in similarity between base and post-trained models (circles and squares). 8/N

1