Lightnews — Scholar-powered news

Valentin Hofmann @valentinhofmann.bsky.social · 9m

Check out this #EMNLP2025 paper led by @minhducbui.bsky.social and @carolin-holtermann.bsky.social showing dialect prejudice remains a major issue in current LLMs.

Example: GPT-5 associates German dialect speakers with being uneducated and steers them toward stereotyped jobs (e.g., farmworkers).

👇

Data Science Hamburg Group @ds-hamburg.bsky.social · 21d

“You speak Bavarian? Then you must be uneducated + closed-minded.”

🤯 Not your opinion? Good. But it might be your LLM’s!!

🧵 Check out our #EMNLP2025 paper, where we uncover concerning dialect bias in recent LLMs - including GPT-5.

#AI #Bias #Dialect #Fairness #LLM #NLProc #Safety

1

Valentin Hofmann @valentinhofmann.bsky.social · 24d

Thanks, Jordan! Your ACL 2021 paper was a huge source of inspiration for us!

1

Valentin Hofmann @valentinhofmann.bsky.social · 24d

We did not specifically analyze novel models as your paper did. While I am optimistic that Fluid Benchmarking improves over static IRT-based methods in this regime as well, there are definitely limitations, which we discuss in the paragraph below.

Would be exciting to run more experiments on this!

Valentin Hofmann @valentinhofmann.bsky.social · 24d

In our experiments, we find that this dynamic approach consistently outperforms static IRT-based methods. The improvements are especially pronounced in terms of variance, which poses a major challenge for static IRT-based methods. We discuss this in more detail in the paragraph below.

1

Valentin Hofmann @valentinhofmann.bsky.social · 24d

Great question! The key difference is that we use IRT to dynamically adapt the subset of items to a model's capability, rather than to determine a static, "globally optimal" subset of items as in prior work. With Fluid Benchmarking, each model is evaluated on a different subset of items.

1

Reposted by Valentin Hofmann

Kyle Lo @kylelo.bsky.social · 26d

LM benchmark design requires 3 decisions, how to:
🐟 select test cases
🐠 score LM on each test
🦈 aggregate scores to estimate perf

fluid benchmarking is simple:
🍣 find max informative test cases
🍥 estimate 'ability', not simple avg perf

why care? turn ur grey noisy benchmarks to red ones!

2 5

Valentin Hofmann @valentinhofmann.bsky.social · 27d

Last but not least, a huge shoutout to my incredible coauthors @davidheineman.com, @ianmagnusson.bsky.social, @kylelo.bsky.social, @jessedodge.bsky.social, @maartensap.bsky.social, Pang Wei Koh, Chun Wang, @hanna-nlp.bsky.social, and @nlpnoah.bsky.social! 🤗

3

Valentin Hofmann @valentinhofmann.bsky.social · 27d

For details, check out our paper, blog, code, and data:

📄 arxiv.org/abs/2509.11106
✍️ allenai.org/blog/fluid-b...
💻 github.com/allenai/flui...
📊 huggingface.co/datasets/all...

Looking forward to chatting more at #COLM2025! 👋

1 2

Valentin Hofmann @valentinhofmann.bsky.social · 27d

Overall, our work shows that LLM evaluations can be substantially improved by moving beyond the until-now universal practice of static benchmarking, which assumes a globally optimal set of evaluation questions for all models.

1 2

Valentin Hofmann @valentinhofmann.bsky.social · 27d

These (and more) advantages are achieved while at the same time reducing evaluation cost.

Example: on MMLU, Fluid Benchmarking results in lower step-to-step variance and higher validity than standard methods while using 50 times fewer questions. ⚡

1 1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

Fluid Benchmarking substantially reduces step-to-step variance during pretraining.

It also increases validity: results generalize better to other benchmarks targeting the same capability. One reason: it automatically avoids mislabeled questions, cutting label errors by 99%! 🤯

1 1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

In our experiments, we apply Fluid Benchmarking to evaluation during pretraining, a setting where capabilities evolve rapidly.

We find that Fluid Benchmarking dynamically adapts to these changes, administering easier questions early in training and more difficult ones later.

1 1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

Fluid Benchmarking repeats this loop until the number of administered questions reaches the allotted budget.

Adaptive question selection means that LLMs face different sets of questions, but ability estimation aligns results in a common space.

1 1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

In Fluid Benchmarking, we start with an initial ability estimate from one question.

To select the next question, we use Fisher information. Essentially: a question close in difficulty (b) to the ability estimate (θ) and with high discrimination (a).

Then we update the estimate.

1 1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

In addition, IRT models each LLM's ability, which can be estimated from its responses to questions with known difficulty and discrimination.

The IRT ability estimate can be used to summarize performance like accuracy, and it accounts for question characteristics.

1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

To get a question's difficulty, we use item response theory (IRT): we analyze responses of hundreds of LLMs to see how often a question is answered correctly.

IRT also measures the discrimination of a question, meaning how reliably it separates stronger from weaker LLMs.

1 1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

Test theory says: questions are most informative when matched to a test taker's ability.

For LLMs, that means evaluating weaker models on easier questions and stronger models on harder ones.

But how do we know a question's difficulty, or an LLM's ability, before evaluation? 🤔

1 1

Valentin Hofmann @valentinhofmann.bsky.social · 27d

📢 New #COLM2025 paper 📢

Standard benchmarks give every LLM the same questions. This is like testing 5th graders and college seniors with *one* exam! 🥴

Meet Fluid Benchmarking, a capability-adaptive eval method delivering lower variance, higher validity, and reduced cost.

🧵

3 10 39

Reposted by Valentin Hofmann

Dallas Card @dallascard.bsky.social · Jul 29

I am delighted to share our new #PNAS paper, with @grvkamath.bsky.social @msonderegger.bsky.social and @sivareddyg.bsky.social, on whether age matters for the adoption of new meanings. That is, as words change meaning, does the rate of adoption vary across generations? www.pnas.org/doi/epdf/10....

3 13 48

Valentin Hofmann @valentinhofmann.bsky.social · Jul 16

Attending #ICML2025? Don't miss this TokShop panel, which will explore:

🔮 The Future of Tokenization 🔮

Featuring a stellar lineup of panelists - mark your calendar! ✨

Tokenization Workshop (TokShop) @ICML2025 @tokshop.bsky.social · Jul 15

🎤 Meet our expert panelists! Join Albert Gu, Alisa Liu, Kris Cao, Sander Land, and Yuval Pinter as they discuss the Future of Tokenization on July 18 at 3:30 PM at TokShop at #ICML2025.

3

Valentin Hofmann @valentinhofmann.bsky.social · Jun 10

LLMs can appear unbiased on the surface but still perpetuate racist views in subtle ways.

What causes this discrepancy? 🔍

In our upcoming #ACL2025 paper, we find a pattern akin to racial colorblindness: LLMs suppress race in ambiguous contexts, leading to biased outcomes.

Lihao Sun @1e0sun.bsky.social · Jun 10

🚨New #ACL2025 paper!

Today’s “safe” language models can look unbiased—but alignment can actually make them more biased implicitly by reducing their sensitivity to race-related associations.

🧵Find out more below!

6

Reposted by Valentin Hofmann

Tokenization Workshop (TokShop) @ICML2025 @tokshop.bsky.social · May 30

📣 We extend the submission deadline by 24 hours to avoid conflict with ACL camera-ready deadline.

📅 New Submission Deadline: May 31, 2025 (23:59 AoE)

📩 OpenReview: openreview.net/group?id=ICM...

1 1

Valentin Hofmann @valentinhofmann.bsky.social · May 29

Huge congrats, Adam!!! 🎉

1

Reposted by Valentin Hofmann

Tokenization Workshop (TokShop) @ICML2025 @tokshop.bsky.social · May 28

Got a good tokenization paper under review at COLM, but the scores were a letdown? 😬

Why bother with rebuttal when the perfect venue is right around the corner!

Submit your paper to the #ICML2025 Tokenization Workshop (TokShop) by May 30! 🚀

4 11

Reposted by Valentin Hofmann

Tokenization Workshop (TokShop) @ICML2025 @tokshop.bsky.social · May 26

Beyond text: Modern AI tokenizes images too! Vision models split photos into patches, treating each 16x16 pixel square as a "token." 🖼️➡️🔤 #VisualTokenization

Interested in tokenization? Join our workshop tokenization-workshop.github.io
The submission deadline is already May 30!

tokenization-workshop.github.io

2 4