Lightnews — Scholar-powered news

Reposted by Vaishnavh Nagarajan

CMU Computer Science Department @csdatcmu.bsky.social · Jul 17

Congratulations to CSD faculty Aditi Raghunathan and her research collaborators on receiving an ICML Outstanding Paper award for Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction (icml.cc/virtual/2025...).

Paper: arxiv.org/abs/2504.15266

ICML 2025 AwardsICML 2025

icml.cc

1 5

Reposted by Vaishnavh Nagarajan

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · Jul 3

Reading the dedications of a PhD thesis is often a cure for a bad day. There’s so much affection in them

4 1 43

Reposted by Vaishnavh Nagarajan

Ahmad Beirami @abeirami.bsky.social · Jul 2

As NeurIPS review deadline is around the corner, please remember that you cannot use any non-local LLM like chatgpt/gemini for understanding the paper and drafting/revising your review as that breaks the confidentiality agreement.

NeurIPS 2025 Official LLM Policy:
neurips.cc/Conferences/...

LLM Policy

neurips.cc

1 6

Reposted by Vaishnavh Nagarajan

Giovanni Toffetti @gtof.bsky.social · Jun 22

I really enjoyed "When We Cease to Understand the World", although it's more fiction than history of science

1 2

Reposted by Vaishnavh Nagarajan

Javier Burroni @jburroni.bsky.social · Jun 23

“Science in history” by Bernal is my first recommendation. The work of Ian Hacking is a good recommendation for
Probability

1 1

Reposted by Vaishnavh Nagarajan

Alexandra Proca @aproca.bsky.social · Jun 20

How do task dynamics impact learning in networks with internal dynamics?

Excited to share our ICML Oral paper on learning dynamics in linear RNNs!
with @clementinedomine.bsky.social @mpshanahan.bsky.social and Pedro Mediano

openreview.net/forum?id=KGO...

Learning dynamics in linear recurrent neural networks

Recurrent neural networks (RNNs) are powerful models used widely in both machine learning and neuroscience to learn tasks with temporal dependencies and to model neural dynamics. However, despite...

openreview.net

1 12 34

Reposted by Vaishnavh Nagarajan

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · Jun 21

When we are doing science, we are unknowingly executing our mythology, taken from movies and friends and textbooks, of what science is. History of science helps us ground that myth in reality

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 12

I finally wrote a full-fledged blog about this: reading the history of science is an **amazing** yet under-recognized way to develop (emotional) maturity as a researcher.

If you have thoughts/recommendations, please share!
vaishnavh.github.io/2025/04/29/h...

3 6 53

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 12

I finally wrote a full-fledged blog about this: reading the history of science is an **amazing** yet under-recognized way to develop (emotional) maturity as a researcher.

If you have thoughts/recommendations, please share!
vaishnavh.github.io/2025/04/29/h...

3 7 35

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

haha that's a new idiom for me. it's perfect! and the flip side is, "target and (potentially) regret", which causes quite a lot of stress. (what if your work gets rejected by the community or worse, overlooked)

1 2

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

but these pressures are real and have always persisted.

I think @abeirami.bsky.social may be interested in this rant.

1 2

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

but now I've the maturity to seek validation from things like "a specific person complimenting my work" or even better, "a meaningful citation where someone substantially builds on my work." (ofc, i also seek internal validation/satisfaction but I gotta be realistic, lol).

1

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

i had intense first-hand struggle with a lot of these effects in my phd since i had <= 1 paper/year for the most part. i started managing it only after getting visibly recognized by experts for one of my papers at one point. i still struggle with it at some level.

1 1

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

then there are many other insidious feedback cycles like the fact that publishing more => more visibility => more opportunities/networks/interfaces with the community/more citations => more opportunities/internships etc., => more papers

2 3

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

for example, with the advent of twitter, there's a pressure to stay constantly visible and to have many different things to say every now and then (bec everyone else is doing that), rather than pitch your one paper again and again which starts feeling awkward :-(

1 4

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

someday I hope to write a blog about "all the other forces that discourage me" from publishing less. people always say "publish less!" but without acknowledging these varied and nuanced forces

1 4

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

all other incentivation strategies I had thought of are much more negative/mean. like
- "evaluating someone based on some bottom k papers" or
- "judging negatively for publishing >N papers"

1 3

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 5

haha thank you! honored you feel that way!

btw, i just noticed this, this sort of a compliment is actually a great way to incentivize people to be more selective in publishing papers (and to counter all the other forces that discourage me from my rate of ~1 paper a year)

2 4

Reposted by Vaishnavh Nagarajan

Tiago Pimentel @tpimentel.bsky.social · Jun 4

A string may get 17 times less probability if tokenised as two symbols (e.g., ⟨he, llo⟩) than as one (e.g., ⟨hello⟩)—by an LM trained from scratch in each situation! Our new ACL paper proposes an observational method to estimate this causal effect! Longer thread soon!

Title of paper "Causal Estimation of Tokenisation Bias" and schematic of how we define tokenisation bias, which is the causal effect we are interested in.

1 9 53

Reposted by Vaishnavh Nagarajan

Andrew Saxe @saxelab.bsky.social · Jun 4

How does in-context learning emerge in attention models during gradient descent training?

Sharing our new Spotlight paper @icmlconf.bsky.social: Training Dynamics of In-Context Learning in Linear Attention
arxiv.org/abs/2501.16265

Led by Yedi Zhang with @aaditya6284.bsky.social and Peter Latham

1 17 52

Reposted by Vaishnavh Nagarajan

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · Jun 2

This paper is quite nice. It mixes some useful toy models of creativity with insights about how to induce more creativity in LLMs that are better than greedy sampling

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 2

📢 New #paper on creativity & multi-token prediction! We design minimal open-ended tasks to argue:

→ LLMs are limited in creativity as they learn to predict the next token

→ creativity can be improved via multi-token learning & injecting noise ("seed-conditioning" 🌱) 1/ #MLSky #AI #arxiv 🧵👇🏽

1 4 16

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 2

Read the full paper: arxiv.org/abs/2504.15266

This work was with great collaborators at CMU: @chenhenrywu.bsky.social who co-led, Charles Ding & @adtraghunathan.bsky.social! Go follow them to see what else they’re up to! 11/

Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

We design a suite of minimal algorithmic tasks that are a loose abstraction of open-ended real-world tasks. This allows us to cleanly and controllably quantify the creative limits of the present-day l...

arxiv.org

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 2

But, there's a lot of scope for exciting work:
→ generalizing these insights to real cows,
→ studying RL/CoT for creativity,
→ understanding surprising behaviors of seed-conditioning 10/👇🏽

1

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 2

Of course, this is all a study of spherical cows. 🐮
Given the noisy, subjective studies of real cows, we believe an objective study brings
→much-needed clarity of thought (like disentangling the two modes of creativity),
→more ideas,
→better-defined experiments. 9/👇🏽

1

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 2

Our vision is that seed-conditioning can help models sample a latent thought and articulate that one thought into words,

but temp sampling has to articulate multiple latent thoughts in parallel to produce a marginal next-word distribution -- this is more burdensome! 8/👇🏽

1 1

Vaishnavh Nagarajan @vaishnavh.bsky.social · Jun 2

Next, we revisit how to produce randomness: the go-to temp sampling 🌡️ vs. injecting a random prefix (seed-conditioning). 🌱

Remarkably, seed-conditioning produces meaningful diversity even w *greedy* decoding 🤑; it is competitive with temp & in some conditions, superior. 7/👇🏽

Figure showing algorithmic creativity with and without seed-conditioning.

1 1