New NeurIPS paper! Why do LMs represent concepts linearly? We focus on LMs's tendency to linearly separate true and false assertions, and provide an analysis of the truth circuit in a toy model. A joint work with Gilad Yehudai, @tallinzen.bsky.social, Joan Bruna and @albertobietti.bsky.social.

✨ Thank you #AISTATS for the best paper award!!

📜 arxiv.org/abs/2410.11067

💡What does VI learn and under what conditions? The answer lies in symmetry.

🤝 Honored to share this award with my co-author Lawrence Saul from @flatironinstitute.org

Come hear Matt Smart's talk about in-context denoising with transformers at the Associative memory workshop #ICLR25, 2:15pm! This task refines the connection between transformers and associative memories. w/ M Smart and @albertobietti.bsky.social at @flatironinstitute.org

Given a high-quality verifier, language model accuracy can be improved by scaling inference-time compute (e.g., w/ repeated sampling). When can we expect similar gains without an external verifier?

New paper: Self-Improvement in Language Models: The Sharpening Mechanism

arxiv.org/abs/2412.01951

📣I'm hiring PhD interns for combined theory+empirical projects in: exploration in post-training, multi-task learning in autoregressive models, distillation, reasoning beyond CoT.

Apply on the link below. If you're at #NeurIPS2024, message me to chat.

jobs.careers.microsoft.com/global/en/jo...

Applications to our Research Fellow position at
@flatironinstitute.org CCM are closing soon on Dec 15! It's a great place for doing fundamental ML research with a lot of freedom, in the heart of NYC. Apply here: apply.interfolio.com/155357

Generating cat videos is nice, but what if you could tackle real scientific problems with the same methods? 🧪🌌
Introducing The Well: 16 datasets (15TB) for Machine Learning, from astrophysics to fluid dynamics and biology.
🐙: github.com/PolymathicAI...
📜: openreview.net/pdf?id=00Sx5...