Definitely impressive and a sign that we have crossed the superhuman threshold in data-rich bounded domains that AI developers care about, such as math! Hard to tell though whether progress will be achievable at the same speed & scale in fuzzier areas, less resources areas

Limited high-quality expert data in narrow domains I guess. I do believe that in this context self-play is the bet that has the potential to go beyond the data curse, but hard to tell whether that will scale to progressively less bounded domains.

I think from current trends it's not unreasonable to think we'll soon cap out at the sharp edges of the jagged capability frontier. Still, there's a long tail of sub-human-level tasks that will see steady progress for years to come.

No notes, matches expectations!

The gap between abstracts and actual submissions was roughly 9k papers for ICML, with ICLR authors being incentivized to submit an abstract because of unreleased decisions. It's probably possible to get to a rough estimate from there!

Great writeup, I thoroughly enjoyed it!

This work aims to contribute to AI safety by developing detection methods, characterizing which architectures are prone to these behaviors, and creating resources for the broader research community. See more in the proposal: sparai.org/projects/sp2...

You can find relevant references in the project description, including excellent work by Transluce, @veraneplenbroek.bsky.social, @arianna-bis.bsky.social @wattenberg.bsky.social , etc. building on recent advances in extracting latent user representations for understanding personalization behaviors.

Mentees will take a leading role in defining research questions, reviewing literature, conducting technical work, adapting codebases, training decoders, or building evaluation pipelines. We'll prioritize 1-2 directions based on your background and interests.

This project aims to expand our understanding of implicit personalization in LLMs: how models form user beliefs, which elements in prompts/training drive these behaviors, and how we can leverage interpretability methods for control beyond simple detection.

When language models interact with users, they implicitly infer user attributes (expertise, demographics, beliefs) that influence responses in ways users neither expect nor endorse. This hidden personalization can lead to sycophancy, deception, and demographic bias.

For more info, see ndif.us, and check out the amazing NNSight toolkit for extracting and analyzing the internals of any Torch-compatible model! nnsight.net