Lightnews — Scholar-powered news

Reposted by Sagnik Mukherjee

ConvAI @ UIUC @convai-uiuc.bsky.social · 19d

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models by @sagnikmukherjee.bsky.social, Lifan Yuan, @dilekh.bsky.social, Hao Peng

Read more here: arxiv.org/abs/2505.11711
x.com/saagnikkk/st...

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Reinforcement learning (RL) yields substantial improvements in large language models (LLMs) downstream task performance and alignment with human values. Surprisingly, such large gains result from upda...

arxiv.org

1 1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

Paper - arxiv.org/abs/2505.11711
Work done with amazing collaborator Lifan Yuan, and advised by our amazing advisors @dilekh.bsky.social and Hao Peng.

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Reinforcement learning (RL) yields substantial improvements in large language models (LLMs) downstream task performance and alignment with human values. Surprisingly, such large gains result from upda...

arxiv.org

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🧵[8/n] To the best of our knowledge this is the first mechanistic evidence that shows contrast between learning from in distribution (or on-policy) data vs Off Distribution (off-policy) data.

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🧵[7/n]

🔍 Potential Reasons

💡 We hypothesize that the in-distribution nature of training data is a key driver behind this sparsity
🧠 The model already "knows" a lot — RL just fine-tunes a small, relevant subnetwork rather than overhauling everything

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🧵[6/n]

🌐 The Subnetwork Is General
🔁 Subnetworks trained with different seed, datasets, or even algorithms show nontrivial overlap
🧩 Suggests the subnetwork is a generalizable structure tied to the base model
🧠 A shared backbone seems to emerge, no matter how you train it

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🧵[5/n]📊
🧪 Training the Subnetwork Reproduces Full Model

1️⃣ When trained in isolation, the sparse subnetwork recovers almost the exact same weights as the full model
2️⃣ achieves comparable (or better) end-task performance
3️⃣ 🧮 Even the training loss converges more smoothly

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🧵[4/n]

📚 Each Layer Is Equally Sparse (or Dense)

📏 No specific layer or sublayer gets special treatment — all layers are updated equally sparsely.
🎯Despite the sparsity, the updates are still full-rank

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🧵[3/n]

📉 Even Gradients Are Sparse in RL 📉

🧠 In PRIME, 72% of parameters never receive any gradient — ever!
↔️ Some do, but their gradients cancel out over time.
🎯 It’s not just sparse updates, even the gradients are sparse

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🧵[2/n]

💡 SFT Updates Are Dense 💡
Unlike RL, Supervised Fine-Tuning (SFT) updates are much denser 🧠
📊 Sparsity is low — at most only 15.31% of parameters remain untouched.

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 21

🚨 Paper Alert: “RL Finetunes Small Subnetworks in Large Language Models”

From DeepSeek V3 Base to DeepSeek R1 Zero, a whopping 86% of parameters were NOT updated during RL training 😮😮
And this isn’t a one-off. The pattern holds across RL algorithms and models.
🧵A Deep Dive

1 3 12

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 7

📂 Code and data coming soon! Read our paper here: arxiv.org/abs/2502.02362

This would not have been possible without the contributions of @abhinav-chinta.bsky.social @takyoung.bsky.social Tarun and our amazing advisor @dilekh.bsky.social Special thanks to the members of @convai-uiuc.bsky.social

1 2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 7

🧠 Additional insights:
1️⃣ Spotting errors in synthetic negative samples is WAY easier than catching real-world mistakes
2️⃣ False positives are inflating math benchmark scores - time for more honest evaluation methods!

🧵[6/n]

1 2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 7

📈 Our results:
PARC improves error detection accuracy by 6-16%, enabling more reliable step-level verification in mathematical reasoning chains.

🧵[5/n]

1 2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 7

📊 The exciting part?
LLMs can reliably identify these critical premises - the specific prior statements that directly support each reasoning step. This creates a transparent structure showing exactly which information is necessary for each conclusion.

🧵[4/n]

1 2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 7

💡 Our solution:
We propose Premise-Augmented Reasoning Chains (PARC): converting linear reasoning into directed graphs by explicitly linking each reasoning step to its necessary premises.
We also propose accumulation errors, an error type ignored in prior work.
🧵[3/n]

1 2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 7

📌 Issue: Verifying lengthy reasoning chains is tough due to hidden step dependencies. The current step doesn’t depend on all previous steps, making the context full of distractors.

🧵[2/n]

1 2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · May 7

🚀Our ICML 2025 paper introduces "Premise-Augmented Reasoning Chains" - a structured approach to induce explicit dependencies in reasoning chains.

By revealing the dependencies within chains, we significantly improve how LLM reasoning can be verified.

🧵[1/n]

1 3 7

Reposted by Sagnik Mukherjee

Dilek Hakkani-Tur @dilekh.bsky.social · Feb 9

AI over-reliance is an important issue for conversational agents. Our work supported mainly by the DARPA FACT program proposes introducing positive friction to encourage users to think critically when making decisions. Great team-work, all!
@convai-uiuc.bsky.social @gokhantur.bsky.social

Mert İnan @merterm.bsky.social · Feb 8

‼️ Ever wish LLMs would just... slow down for a second?

In our latest work, "Better Slow than Sorry: Introducing Positive Friction for Reliable Dialogue Systems", we delve into how strategic delays can enhance dialogue systems.

Paper Website: merterm.github.io/positive-fri...

3 10

Reposted by Sagnik Mukherjee

Marc Marone @marcmarone.com · Nov 23

I noticed a lot of starter packs skewed towards faculty/industry, so I made one of just NLP & ML students: go.bsky.app/vju2ux

Students do different research, go on the job market, and recruit other students. Ping me and I'll add you!

100 54 180

Sagnik Mukherjee @sagnikmukherjee.bsky.social · Nov 24

🙋

2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · Nov 21

Work done with an amazing team (most of them are not here yet, other than @faridlazuarda.bsky.social )

2

Sagnik Mukherjee @sagnikmukherjee.bsky.social · Nov 21

We call out that most studies of culture has focused on a "thin" description.
"Digitally under-represented cultures are more likely to get represented by their “thin descriptions" created by “outsiders" on the digital space, which can further aggravate the biases and stereotypes."

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · Nov 21

🚩 We discovered some key gaps: Incomplete cultural coverage, issues with methodological robustness, and a lack of situated studies for real-world applicability. These gaps limit our understanding of cultural biases in LLMs. [5/7]

1

Sagnik Mukherjee @sagnikmukherjee.bsky.social · Nov 21

📚 Most studies use black-box probing methods to examine LLMs' cultural biases. However, these methods can be sensitive to prompt wording, raising concerns about robustness and generalizability. [4/7]

1