1/🚨 New preprint

How do #LLMs’ inner features change as they train? Using #crosscoders + a new causal metric, we map when features appear, strengthen, or fade across checkpoints—opening a new lens on training dynamics beyond loss curves & benchmarks.

#interpretability

Paper: arxiv.org/pdf/2508.04796
Code: github.com/swiss-ai/par...

In short, Parity-aware BPE=minimal overhead+clear fairness gains. If you care about multilingual robustness, tokenization is low-hanging fruit.
Joint work with Clara Meister, @debjit-paul.bsky.social @joelniklaus.bsky.social @sinaahmadi.bsky.social @abosselut.bsky.social @ricosennrich.bsky.social

What’s even more exciting: low- and medium-resource languages benefit the most. We see better vocabulary utilization and compression rates for these languages, highlighting the effectiveness of our approach in providing fairer language allocation.

Empirical results: Gini coefficient of tokenizer disparity (0 indicates a tokenizer's compression rates across languages are equal) improves by ~83% with global compression remaining very similar. On downstream task accuracy, improvements outnumber declines across configurations

It’s a drop-in replacement in existing systems that introduces minimal training-time overhead: if you already use a BPE tokenizer, formats and tokenization/detokenization at inference are unchanged. You just need language-labeled multilingual corpora and a multi-parallel dev set.

What changes from classical BPE? Only a small part of training. We compute frequency stats per language → when choosing the next merge, we pick it from the stats of the language with the worst compression rate, rather than from global stats. Everything else stays the same!

🚨New Preprint!

In multilingual models, the same meaning can take far more tokens in some languages, penalizing users of underrepresented languages with worse performance and higher API costs. Our Parity-aware BPE algorithm is a step toward addressing this issue: 🧵

Stop by our poster presentation at @iclr-conf.bsky.social and discuss real multilingual evaluation!
Feel free to reach out anytime during the conference! We’d love to connect!

NEW PAPER ALERT: Generating visual narratives to illustrate textual stories remains an open challenge, due to the lack of knowledge to constrain faithful and self-consistent generations. Our #CVPR2025 paper proposes a new benchmark, VinaBench, to address this challenge.

Lots of great news out of the EPFL NLP lab these last few weeks. We'll be at @iclr-conf.bsky.social and @naaclmeeting.bsky.social in April / May to present some of our work in training dynamics, model representations, reasoning, and AI democratization. Come chat with us during the conference!

🚀 Introducing PICLe: a framework for in-context named-entity detection (NED) using pseudo-annotated demonstrations.
🎯 No human labeling needed—yet it outperforms few-shot learning with human annotations!
#AI #NLProc #LLMs #ICL #NER

What’s your take on integrating AI into education while maintaining rigor? 🤔
Check out the paper for the key findings and join the discussion on AI’s place in higher education: t.co/tJ8Gg1FRCy

AI is reshaping #education, but are we ready? 🚨
Our new
@pnas.org
article explores how #LLMs challenge traditional assessments in higher education.
Instead of banning #AI, we argue for redesigning assessments to emphasize real-world problem-solving and ethical AI use.

INCLUDE evaluates how well LLMs grasp regional knowledge—local customs, culture, and info users actually need.
With ~200K questions from 52 countries, it's time to build AI that truly includes 🤗

📄Check out our paper for more details:
arxiv.org/abs/2411.19799

Excited to share our work on INCLUDE! 🚀
INCLUDE sets a new standard for #LLM benchmarks—spanning 44 languages with a focus on regional knowledge and cultural context 🌍
Time for LLMs to meet the world where it is, not where it’s translated to!
#Multilingual #AI #NLProc

[email protected] is hiring for multiple positions in CS (including one open call): www.epfl.ch/about/workin...

Apply to come join us in Beautiful Lausanne!

EPFL's new AI Center has a Call for applications for postdoc fellowships in all AI-related areas. Come join if you're interested in working with me and fantastic AI colleagues!

Extra Perk: We actually do have lots of GPUs !

Deadline: November 29th

More info at:
www.epfl.ch/research/fun...

✋🏻

✋🏻

✋🏻

✋🏻