Lightnews — Scholar-powered news

Reposted by Sheridan Feucht @ COLM

Kanishka Misra 🌊 @kanishka.bsky.social · Jul 28

Looking forward to attending #cogsci2025 (Jul 29 - Aug 3)! I’m especially excited to meet students who will be applying to PhD programs in Computational Ling/CogSci in the coming cycle.

Please reach out if you want to meet up and chat! Email is the best way, but DM also works if you must!

quick🧵:

Placeholders for 3 students (number arbitrarily chosen) and me - to signify my eventual group!

1 7 21

Sheridan Feucht @ COLM @sfeucht.bsky.social · Jul 22

Try it out in our new paper demo notebook! Or ping me with any sequence to try and I'd be more than happy to run a few examples for you.
colab.research.google.com/github/sfeuc...

Also check out the new camera-ready version of the paper on arXiv.
arxiv.org/abs/2504.03022

Google Colab

colab.research.google.com

1

Sheridan Feucht @ COLM @sfeucht.bsky.social · Jul 22

If we do the same for token induction heads, we can also get a "token lens", which reads out surface-level token information from states. Unlike raw logit lens, which reveals next-token predictions, "token lens" reveals the current token.

"Token lens" outputs for the token "card" in the context "in the morning air, she heard northern card.inals."

1

Sheridan Feucht @ COLM @sfeucht.bsky.social · Jul 22

If we apply concept lens to the word "cardinals" in three contexts, we see that Llama-2-7b has encoded this word very differently in each case!

Three "concept lens" outputs, showing the top-5 highest probability tokens when a hidden state (throughout different layers) is transformed by concept lens and projected to token space. There are three sentences, each with different predictions: "he was a lifelong fan of the cardinals", for which concept lens predicts "football" and "baseball"; "the secret meeting of the cardinals", for which concept lens predicts "Catholic"; and "in the morning air, she hear northern cardinals", which projects to "birds."

1 1

Sheridan Feucht @ COLM @sfeucht.bsky.social · Jul 22

To do this, we sum the OV matrices of the top-k concept induction heads, and use it to transform a hidden state at a particular token position. Projecting that to vocab space with the model's decoder head, we can access the "meaning" encoded in that state.

1

Sheridan Feucht @ COLM @sfeucht.bsky.social · Jul 22

We've added a quick new section to this paper, which was just accepted to @COLM_conf! By summing weights of concept induction heads, we created a "concept lens" that lets you read out semantic information in a model's hidden states. 🔎

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

[📄] Are LLMs mindless token-shifters, or do they build meaningful representations of language? We study how LLMs copy text in-context, and physically separate out two types of induction heads: token heads, which copy literal tokens, and concept heads, which copy word meanings.

1 1 7

Reposted by Sheridan Feucht @ COLM

Koyena Pal @koyena.bsky.social · Jun 30

🚨 Registration is live! 🚨

The New England Mechanistic Interpretability (NEMI) Workshop is happening Aug 22nd 2025 at Northeastern University!

A chance for the mech interp community to nerd out on how models really work 🧠🤖

🌐 Info: nemiconf.github.io/summer25/
📝 Register: forms.gle/v4kJCweE3UUH...

8 10

Sheridan Feucht @ COLM @sfeucht.bsky.social · Jun 24

Nikhil's recent paper is a tour de force in causal analysis! They show that LLMs keep track of what characters know in a story using "pointer" mechanisms. Definitely worth checking out.

nikhil07prakash.bsky.social @nikhil07prakash.bsky.social · Jun 24

How do language models track mental states of each character in a story, often referred to as Theory of Mind?

We reverse-engineered how LLaMA-3-70B-Instruct handles a belief-tracking task and found something surprising: it uses mechanisms strikingly similar to pointer variables in C programming!

2 4

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 25

I'm on the train right now and just finished reading this paper for the first time--I actually just logged back on to bsky just so that I could link to it, but you beat me to the punch!

I really enjoyed your paper. This example was particularly great.

1

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 25

I used to think formal reasoning was central to language and intelligence, but now I’m not so sure. Wrote a short post about my thoughts on this, with a couple chewy anecdotes. Would love to get some feedback or pointers to further reading.
sfeucht.github.io/syllogisms/

Sheridan Feucht

Solving Syllogisms is Not Intelligence April 23, 2025 (I think that we overvalue logical reasoning when it comes to measuring "intelligence.") What do we mean by intelligence in the context of cogniti...

sfeucht.github.io

1 5

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 10

I'll present a poster for this work at NENLP tomorrow! Come find me at poster #80...

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

[📄] Are LLMs mindless token-shifters, or do they build meaningful representations of language? We study how LLMs copy text in-context, and physically separate out two types of induction heads: token heads, which copy literal tokens, and concept heads, which copy word meanings.

1 7

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 9

That’s a good point! Sort of related, I noticed last night that when I have to type in a 2FA code I usually compress the numbers. Like if the code is 51692 I think “fifty-one, sixty-nine, two.” I wonder if this is a thing that people have studied. Thanks for the comment :)

1

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

Paper: arxiv.org/abs/2504.03022
Code: github.com/sfeucht/dual...

See dualroute.baulab.info for more info. Work done with @ericwtodd.bsky.social, @byron.bsky.social, and @davidbau.bsky.social. :)

The Dual-Route Model of Induction

Prior work on in-context copying has shown the existence of induction heads, which attend to and promote individual tokens during copying. In this work we introduce a new type of induction head: conce...

arxiv.org

2

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

Yin & Steinhardt (2025) recently showed that FV heads are more important for ICL than token induction heads. But for translation, *concept* induction heads matter too! They copy forward word meanings, whereas FV heads influence the output language.
bsky.app/profile/kay...

1 2

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

Concept heads also output language-agnostic word representations. If we patch the outputs of these heads from one translation prompt to another, we can change the *meaning* of the outputted word, without changing the language. (see prior work from @butanium.bsky.social and @wendlerc.bsky.social)

1 4

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

Token induction heads are still important, though. When we ablate them over long sequences, models start to paraphrase instead of copying. We take this to mean that token induction heads are responsible for *exact* copying (which concept induction heads apparently can't do).

1 2

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

But how do we know these heads copy semantics? When we ablate concept induction heads, performance drops drastically for translation, synonyms, and antonyms: all tasks that require copying *meaning*, not just literal tokens.

1 3

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

Previous work showed that token induction heads attend to the next token to be copied (*window*pane). Analogously, we find that concept induction heads attend to the end of the next multi-token word to be copied (windowp*ane*).

1 2

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

--using causal interventions. Essentially, we pick out all of the attention heads that are responsible for promoting future entity tokens (e.g. "ax" in "waxwing"). We hypothesize that heads carrying an entire entity actually represent the *meaning* of that chunk of tokens.

1 2

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

Induction heads were discovered by Elhage et al. (2021) and Olsson et al. (2022). They focused on token copying, but some of the heads they found also seemed to activate for "fuzzy" copying tasks, like translation. We directly identify these heads--
transformer-circuits.pub/2022/in-con...

1 3

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

There are multiple ways to copy text! Copying a wifi password like hxioW2qN52 is different than copying a meaningful one like OwlDoorGlass. Nonsense copying requires each char to be transferred one-by-one, but meaningful words can be copied all at once. Turns out, LLMs do both.

2 1 5

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 7

[📄] Are LLMs mindless token-shifters, or do they build meaningful representations of language? We study how LLMs copy text in-context, and physically separate out two types of induction heads: token heads, which copy literal tokens, and concept heads, which copy word meanings.

1 20 77

Sheridan Feucht @ COLM @sfeucht.bsky.social · Apr 1

So gorgeous, is this in Cambridge?

1 1

Sheridan Feucht @ COLM @sfeucht.bsky.social · Mar 12

Looks really cool! Can’t wait to give this a proper read.

5

Reposted by Sheridan Feucht @ COLM

Chantal @chantalsh.bsky.social · Mar 10

I'm searching for some comp/ling experts to provide a precise definition of “slop” as it refers to text (see: corp.oup.com/word-of-the-...)

I put together a google form that should take no longer than 10 minutes to complete: forms.gle/oWxsCScW3dJU...
If you can help, I'd appreciate your input! 🙏

Oxford Word of the Year 2024 - Oxford University Press

The Oxford Word of the Year 2024 is 'brain rot'. Discover more about the winner, our shortlist, and 20 years of words that reflect the world.

corp.oup.com

8 10