megamor2.bsky.social
Mor Geva
@megamor2.bsky.social
840 followers 76 following 29 posts
https://mega002.github.io
mega002.github.io
Posts Media Videos Starter Packs
Reposted by Mor Geva
yoav.ml
Yoav Gur Arieh @yoav.ml · 13h
🧠 To reason over text and track entities, we find that language models use three types of 'pointers'!

They were thought to rely only on a positional one—but when many entities appear, that system breaks down.

Our new paper shows what these pointers are and how they interact 👇
1 1 3
Reposted by Mor Geva
soheeyang.bsky.social
Sohee Yang @soheeyang.bsky.social · Jun 13
🚨 New Paper 🚨
How effectively do reasoning models reevaluate their thought? We find that:
- Models excel at identifying unhelpful thoughts but struggle to recover from them
- Smaller models can be more robust
- Self-reevaluation ability is far from true meta-cognitive awareness
1/N 🧵
1 3 12
Reposted by Mor Geva
yoav.ml
Yoav Gur Arieh @yoav.ml · May 29
New Paper Alert! Can we precisely erase conceptual knowledge from LLM parameters?
Most methods are shallow, coarse, or overreach, adversely affecting related or general knowledge.

We introduce🪝𝐏𝐈𝐒𝐂𝐄𝐒 — a general framework for Precise In-parameter Concept EraSure. 🧵 1/
1 2 5
Reposted by Mor Geva
mariusmosbach.bsky.social
Marius Mosbach @mariusmosbach.bsky.social · Apr 15
Checkout Benno's notes about our impact of interpretability paper 👇.

Also, we are organizing a workshop at #ICML2025 which is inspired by some of the questions discussed in the paper: actionable-interpretability.github.io
3 11
Reposted by Mor Geva
sarah-nlp.bsky.social
Sarah Wiegreffe @sarah-nlp.bsky.social · Apr 3
Have work on the actionable impact of interpretability findings? Consider submitting to our Actionable Interpretability workshop at ICML! See below for more info.

Website: actionable-interpretability.github.io
Deadline: May 9
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Mar 31
🎉 Our Actionable Interpretability workshop has been accepted to #ICML2025! 🎉
> Follow @actinterp.bsky.social
> Website actionable-interpretability.github.io

@talhaklay.bsky.social @anja.re @mariusmosbach.bsky.social @sarah-nlp.bsky.social @iftenney.bsky.social

Paper submission deadline: May 9th!
10 20
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Mar 31
Forgot to tag the one and only @hadasorgad.bsky.social !!!
2
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Mar 31
🎉 Our Actionable Interpretability workshop has been accepted to #ICML2025! 🎉
> Follow @actinterp.bsky.social
> Website actionable-interpretability.github.io

@talhaklay.bsky.social @anja.re @mariusmosbach.bsky.social @sarah-nlp.bsky.social @iftenney.bsky.social

Paper submission deadline: May 9th!
3 16 43
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Feb 24
📣 📣 Looking for ethics reviewers for COLM 2025!
Please sign up and share the form below 👇
forms.gle/3a52jbDNB9bd...
COLM 2025 Ethics Reviewer Sign Up
Ethics reviewing of papers for COLM 2025 starts in May. We will share more details later. In the meantime, please sign up.
forms.gle
2 2
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Feb 13
Communication between LLM agents can be super noisy! One rogue agent can easily drag the whole system into failure 😱

We find that (1) it's possible to detect rogue agents early on
(2) interventions can boost system performance by up to 20%!

Thread with details and paper link below!
ohav.bsky.social
ohav.bsky.social @ohav.bsky.social · Feb 13
"One bad apple can spoil the bunch 🍎", and that's doubly true for language agents!
Our new paper shows how monitoring and intervention can prevent agents from going rogue, boosting performance by up to 20%. We're also releasing a new multi-agent environment 🕵️‍♂️
5
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
Check out our paper and code for more details, analyses, and cool examples!

🔗 Paper: arxiv.org/abs/2501.08319
🔗 HF: huggingface.co/papers/2501.08319
🔗 Code: github.com/yoavgur/Feature-Descriptions

7/
Enhancing Automated Interpretability with Output-Centric Feature Descriptions
Automated interpretability pipelines generate natural language descriptions for the concepts represented by features in large language models (LLMs), such as plants or the first word in a sentence. Th...
arxiv.org
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
In a final experiment, we show that output-centric methods can be used to "revive" features previously thought to be "dead" 🧟‍♂️ reviving hundreds of SAE features in Gemma 2! 6/
1
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
In a final experiment, we show that output-centric methods can be used to "revive" features previously thought to be "dead" 🧟‍♂️ reviving hundreds of SAE features in Gemma 2! 6/
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
Unsurprisingly, while activating inputs better describe what activates a feature, output-centric methods do much better at predicting how steering the feature will affect the model’s output!

But combining the two works best! 🚀 5/
2
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
Next, we evaluate the widely-used activating inputs approach versus two output-centric methods:
- vocabulary projection (a.k.a logit lens)
- tokens with max probability change in the output

Our output-centric methods require no more than a few inference passes! 4/
1
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
To fix this, we first propose using both input- and output-based evaluations for feature descriptions.
Our output-based eval measures how well a description of a feature captures its effect on the model's generation. 3/
1
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
Autointerp pipelines describe neurons and SAE features based on inputs that activate them.

This is problematic ⚠️
1. Collecting activations for large data is expensive, time-consuming, and often unfeasible.
2. It overlooks how features affect model outputs!

2/
1
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Jan 28
How can we interpret LLM features at scale? 🤔

Current pipelines use activating inputs, which is costly and ignores how features causally affect model outputs!
We propose efficient output-centric methods that better predict the steering effect of a feature.

New preprint led by @yoav.ml 🧵1/
1 4 34
Reposted by Mor Geva
fbarez.bsky.social
Fazl Barez @fbarez.bsky.social · Jan 10

🚨 New Paper Alert: Open Problem in Machine Unlearning for AI Safety 🚨

Can AI truly "forget"? While unlearning promises data removal, controlling emergent capabilities is a inherent challenge. Here's why it matters: 👇

Paper: arxiv.org/pdf/2501.04952
1/8
1 6 25
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Dec 18
Check out our paper and code for more details and cool results!
Paper: arxiv.org/abs/2412.11965
Code: github.com/amitelhelo/M...

(10/10!)
Inferring Functionality of Attention Heads from their Parameters
Attention heads are one of the building blocks of large language models (LLMs). Prior work on investigating their operation mostly focused on analyzing their behavior during inference for specific cir...
arxiv.org
1 2
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Dec 18
Most operation descriptions are plausible based on human judgment.
We also observe interesting operations implemented by heads, like the extension of time periods (day → month → year) and association of known figures with years relevant to their historical significance (9/10)
1 1
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Dec 18
Next, we establish an automatic pipeline that uses GPT-4o to annotate the salient mappings from MAPS.
We map the attention heads of Pythia 6.9B and GPT2-xl and manage to identify operations for most heads, reaching 60%-96% in the middle and upper layers (8/10)
1 2
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Dec 18
(3) Smaller models tend to encode higher numbers of relations in a single head

(4) In Llama-3.1 models, which use grouped-query attention, grouped heads often implement the same or similar relations (7/10)
1 2
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Dec 18
(1) Different models encode certain relations across attention heads to similar degrees

(2) Different heads implement the same relation to varying degrees, which has implications for localization and editing of LLMs (6/10)
1 3
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Dec 18
Using MAPS, we study the distribution of operations across heads in different models -- Llama, Pythia, Phi, GPT2 -- and see some cool trends of function encoding universality and architecture biases: (5/10)
1
megamor2.bsky.social
Mor Geva @megamor2.bsky.social · Dec 18
Experiments on 20 operations and 6 LLMs show that MAPS estimations strongly correlate with the head’s outputs during inference

Ablating heads implementing an operation damages the model’s ability to perform tasks requiring the operation compared to removing other heads (4/10)
1 3