Lightnews — Scholar-powered news

Reposted by Angelika Romanou

Deniz Bayazit @bayazitdeniz.bsky.social · 13d

1/🚨 New preprint

How do #LLMs’ inner features change as they train? Using #crosscoders + a new causal metric, we map when features appear, strengthen, or fade across checkpoints—opening a new lens on training dynamics beyond loss curves & benchmarks.

#interpretability

2 6 13

Angelika Romanou @agromanou.bsky.social · 26d

Proud to have been part of the team behind #Apertus 🌍✨ an open multilingual LLM.

Trained on open data, supporting 1,800+ languages, and built with transparency, compliance & responsible AI in mind.

🤖 Try Apertus models: huggingface.co/collections/...

1

Angelika Romanou @agromanou.bsky.social · Apr 23

If you’re at @iclr-conf.bsky.social this week, come check out our spotlight poster INCLUDE during the Thursday 3:00–5:30pm session!

I will be there to chat about all things multilingual & multicultural evaluation.

Feel free to reach out anytime during the conference. I’d love to connect!

Angelika Romanou @agromanou.bsky.social · Dec 2

🚀 Introducing INCLUDE 🌍: A multilingual LLM evaluation benchmark spanning 44 languages!

Contains *newly-collected* data, prioritizing *regional knowledge*.
Setting the stage for truly global AI evaluation.
Ready to see how your model measures up?
#AI #Multilingual #LLM #NLProc

2 4

Reposted by Angelika Romanou

silingao.bsky.social @silingao.bsky.social · Apr 1

NEW PAPER ALERT: Generating visual narratives to illustrate textual stories remains an open challenge, due to the lack of knowledge to constrain faithful and self-consistent generations. Our #CVPR2025 paper proposes a new benchmark, VinaBench, to address this challenge.

1 5 6

Reposted by Angelika Romanou

Antoine Bosselut @abosselut.bsky.social · Feb 25

Lots of great news out of the EPFL NLP lab these last few weeks. We'll be at @iclr-conf.bsky.social and @naaclmeeting.bsky.social in April / May to present some of our work in training dynamics, model representations, reasoning, and AI democratization. Come chat with us during the conference!

1 12 25

Reposted by Angelika Romanou

Badr AlKhamissi @bkhmsi.bsky.social · Dec 19

🚨 New Paper!

Can neuroscience localizers uncover brain-like functional specializations in LLMs? 🧠🤖

Yes! We analyzed 18 LLMs and found units mirroring the brain's language, theory of mind, and multiple demand networks!

w/ @gretatuckute.bsky.social, @abosselut.bsky.social, @mschrimpf.bsky.social
🧵👇

2 27 110

Reposted by Angelika Romanou

Sepideh Mamooler@ACL🇦🇹 @smamooler.bsky.social · Dec 17

🚀 Introducing PICLe: a framework for in-context named-entity detection (NED) using pseudo-annotated demonstrations.
🎯 No human labeling needed—yet it outperforms few-shot learning with human annotations!
#AI #NLProc #LLMs #ICL #NER

1 8 12

Angelika Romanou @agromanou.bsky.social · Dec 5

Introducing Global-MMLU🌍: A multilingual benchmark featuring MMLU translations in 42 languages crafted with:
✅ Human curation
✅ Extensive metadata
✅ Insights into cultural sensitivity

Proud to have collaborated with Shivalika Singh, @sarahooker.bsky.social and Cohere For AI!

Sara Hooker @sarahooker.bsky.social · Dec 5

Is MMLU Western-centric? 🤔

As part of a massive cross-institutional collaboration:
🗽Find MMLU is heavily overfit to western culture
🔍 Professional annotation of cultural sensitivity data
🌍 Release improved Global-MMLU 42 languages

📜 Paper: arxiv.org/pdf/2412.03304
📂 Data: hf.co/datasets/Coh...

4 13

Reposted by Angelika Romanou

Antoine Bosselut @abosselut.bsky.social · Dec 4

1/ 📘 Could ChatGPT get an engineering degree? Spoiler, yes! In our new @pnas.org article, we explore how AI assistants like GPT-4 perform in STEM university courses — and on average they pass a staggering 91.7% of core courses. 🧵 #AI #HigherEd #STEM #LLMs #NLProc

1 14 36

Angelika Romanou @agromanou.bsky.social · Dec 2

🙋🏻‍♀️

Angelika Romanou @agromanou.bsky.social · Dec 2

👏 As well as the fantastic multilingual research community that helped us collect and validate INCLUDE!

3

Angelika Romanou @agromanou.bsky.social · Dec 2

🙏 We thank our amazing core team and advisors:
@negarforoutan.bsky.social, Anna Sotnikova, @eric-zemingchen.bsky.social, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Imanol Schlag, @mziizm.bsky.social, @sarahooker.bsky.social, @abosselut.bsky.social

1 4

Angelika Romanou @agromanou.bsky.social · Dec 2

For easy evaluation, we provide the following subsets:
INCLUDE-base: up to 550 samples per language, totaling ~23K questions
🤗 : huggingface.co/datasets/Coh...
INCLUDE-lite: up to 250 samples per language, totaling ~11K questions
🤗 : huggingface.co/datasets/Coh...

1 5

Angelika Romanou @agromanou.bsky.social · Dec 2

Check out our paper: 📄 : arxiv.org/abs/2411.19799

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI ...

arxiv.org

1 4

Angelika Romanou @agromanou.bsky.social · Dec 2

🤝 Information is transferred across languages of the same script, though untrained languages might also excel due to potential data contamination.

🌎 Models can struggle with non-English instructions, entangling knowledge evaluation with other factors such as task formatting.

1 3

Angelika Romanou @agromanou.bsky.social · Dec 2

Analysis shows:
📚 Models have a long way to go in capturing the regional knowledge reflected in languages.

💪 Model scale improves regional knowledge understanding, but other techniques like CoT or instruction tuning have minimal or negative impacts.

1 3

Angelika Romanou @agromanou.bsky.social · Dec 2

To build INCLUDE, we collected ~200K MCQ data from 44 languages and 58 knowledge domains, collected from local sources in 52 countries, representing a rich array of cultural and regional knowledge.

1 1 6

Angelika Romanou @agromanou.bsky.social · Dec 2

🤔 Why is regional knowledge so important?

Users expect #LLMs to know information relevant to their environments— customs, culture, etc.
To be relevant & relatable, LLMs need to know these nuances. It's not just global knowledge; it's about meeting user needs where they are.

1 1 4

Angelika Romanou @agromanou.bsky.social · Dec 2

🌍 First, what is regional knowledge?

It's the local info, culture & practices of a regional context. US Law is a great topic, but not as relevant for multilingual LLMs for other regions.

For INCLUDE, we collect regional knowledge rather than translating Western-centric benchmarks.

1 4

Angelika Romanou @agromanou.bsky.social · Dec 2

🚀 Introducing INCLUDE 🌍: A multilingual LLM evaluation benchmark spanning 44 languages!

Contains *newly-collected* data, prioritizing *regional knowledge*.
Setting the stage for truly global AI evaluation.
Ready to see how your model measures up?
#AI #Multilingual #LLM #NLProc

1 6 36

Angelika Romanou @agromanou.bsky.social · Nov 26

🙋🏻‍♀️

Reposted by Angelika Romanou

Antoine Bosselut @abosselut.bsky.social · Nov 26

[email protected] is hiring for multiple positions in CS (including one open call): www.epfl.ch/about/workin...

Apply to come join us in Beautiful Lausanne!

Open Faculty Positions

-

www.epfl.ch

9 13

Angelika Romanou @agromanou.bsky.social · Nov 23

🙋🏻‍♀️

1 1

Angelika Romanou @agromanou.bsky.social · Nov 22

🙋🏻‍♀️

1

Angelika Romanou @agromanou.bsky.social · Nov 21

🙋🏻‍♀️

1 1