Chaitanya Malaviya
@cmalaviya.bsky.social
Senior research scientist @ GoogleDeepMind | benchmarking and evaluation | prev @upenn.edu @ai2.bsky.social, and @ltiatcmu.bsky.social
chaitanyamalaviya.github.io
chaitanyamalaviya.github.io
Pinned
Ever wondered what makes language models generate overly verbose, vague, or sycophantic responses?
Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
Reposted by Chaitanya Malaviya
LLMs power research, decision‑making, and exploration—but most benchmarks don’t test how well they stitch together evidence across dozens (or hundreds) of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning. 👇
August 18, 2025 at 4:05 PM
LLMs power research, decision‑making, and exploration—but most benchmarks don’t test how well they stitch together evidence across dozens (or hundreds) of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning. 👇
People at #ACL2025, come drop by our poster today & chat with me about how context matters for reliable language model evaluations!
Jul 30, 11:00-12:30 at Hall 4X, board 424.
Jul 30, 11:00-12:30 at Hall 4X, board 424.
Excited to share ✨ Contextualized Evaluations ✨!
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
July 30, 2025 at 6:05 AM
People at #ACL2025, come drop by our poster today & chat with me about how context matters for reliable language model evaluations!
Jul 30, 11:00-12:30 at Hall 4X, board 424.
Jul 30, 11:00-12:30 at Hall 4X, board 424.
Reposted by Chaitanya Malaviya
issues w preference LM benchmarks:
🐡data contains cases where the "bad" response is just as good as chosen one
🐟model rankings can feel off (claude ranks lower than expected)
led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025
🐡data contains cases where the "bad" response is just as good as chosen one
🐟model rankings can feel off (claude ranks lower than expected)
led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025
In our new paper, “Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries,” we find that adding just a bit of missing context can reorder model leaderboards—and surface hidden biases. 🧵👇
July 22, 2025 at 5:02 PM
issues w preference LM benchmarks:
🐡data contains cases where the "bad" response is just as good as chosen one
🐟model rankings can feel off (claude ranks lower than expected)
led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025
🐡data contains cases where the "bad" response is just as good as chosen one
🐟model rankings can feel off (claude ranks lower than expected)
led by @cmalaviya.bsky.social, we study underspecified queries & detrimental effect on model evals; accepted to TACL 2025
Context is an overlooked aspect of language model evaluations. Check out how to incorporate context into evaluations in our TACL paper, how it changes evaluation conclusions and makes evaluation more reliable!
In our new paper, “Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries,” we find that adding just a bit of missing context can reorder model leaderboards—and surface hidden biases. 🧵👇
July 22, 2025 at 5:12 PM
Context is an overlooked aspect of language model evaluations. Check out how to incorporate context into evaluations in our TACL paper, how it changes evaluation conclusions and makes evaluation more reliable!
Check out our paper and counterfactual data below! 👇
• Paper: arxiv.org/abs/2506.05339
• Data: huggingface.co/datasets/abh...
• Code: github.com/anirudhb123/...
• Paper: arxiv.org/abs/2506.05339
• Data: huggingface.co/datasets/abh...
• Code: github.com/anirudhb123/...
Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
Language models serve as proxies for human preference judgements in alignment and evaluation, yet they exhibit systematic miscalibration, prioritizing superficial patterns over substantive qualities. ...
arxiv.org
June 6, 2025 at 4:32 PM
Check out our paper and counterfactual data below! 👇
• Paper: arxiv.org/abs/2506.05339
• Data: huggingface.co/datasets/abh...
• Code: github.com/anirudhb123/...
• Paper: arxiv.org/abs/2506.05339
• Data: huggingface.co/datasets/abh...
• Code: github.com/anirudhb123/...
Our findings suggest that targeted debiasing using counterfactuals can help build more reliable preference models, a key step for both LLM alignment and evaluation.
Work led by Anirudh and done jointly with Nitish and @yatskar.bsky.social .
Work led by Anirudh and done jointly with Nitish and @yatskar.bsky.social .
June 6, 2025 at 4:32 PM
Our findings suggest that targeted debiasing using counterfactuals can help build more reliable preference models, a key step for both LLM alignment and evaluation.
Work led by Anirudh and done jointly with Nitish and @yatskar.bsky.social .
Work led by Anirudh and done jointly with Nitish and @yatskar.bsky.social .
For instance, miscalibration for vagueness dropped from 51.3% to 28.5% and for jargon from 50.3% to 33.2% after CDA.
Even joint debiasing across multiple biases (length, vagueness, jargon) proved effective with minimal impact on general capabilities.
Even joint debiasing across multiple biases (length, vagueness, jargon) proved effective with minimal impact on general capabilities.
June 6, 2025 at 4:32 PM
For instance, miscalibration for vagueness dropped from 51.3% to 28.5% and for jargon from 50.3% to 33.2% after CDA.
Even joint debiasing across multiple biases (length, vagueness, jargon) proved effective with minimal impact on general capabilities.
Even joint debiasing across multiple biases (length, vagueness, jargon) proved effective with minimal impact on general capabilities.
And the results? CDA works!
It significantly reduced average miscalibration (e.g., from 39.4% to 32.5%) and brought model skew much closer to human preferences. All this while maintaining overall performance on RewardBench!
It significantly reduced average miscalibration (e.g., from 39.4% to 32.5%) and brought model skew much closer to human preferences. All this while maintaining overall performance on RewardBench!
June 6, 2025 at 4:32 PM
And the results? CDA works!
It significantly reduced average miscalibration (e.g., from 39.4% to 32.5%) and brought model skew much closer to human preferences. All this while maintaining overall performance on RewardBench!
It significantly reduced average miscalibration (e.g., from 39.4% to 32.5%) and brought model skew much closer to human preferences. All this while maintaining overall performance on RewardBench!
So how do we debias models? We propose a simple yet effective post-training method based on counterfactual data augmentation (CDA).
We synthesize contrastive responses that explicitly magnify biases in dispreferred responses, & further finetune reward models on these responses.
We synthesize contrastive responses that explicitly magnify biases in dispreferred responses, & further finetune reward models on these responses.
June 6, 2025 at 4:32 PM
So how do we debias models? We propose a simple yet effective post-training method based on counterfactual data augmentation (CDA).
We synthesize contrastive responses that explicitly magnify biases in dispreferred responses, & further finetune reward models on these responses.
We synthesize contrastive responses that explicitly magnify biases in dispreferred responses, & further finetune reward models on these responses.
Indeed, preference models can easily latch on to these subtle data artifacts!
Features that only weakly correlate with human preferences (r_human=−0.12) are strongly predictive for models (r_model=0.36). Points above y=x suggest that models overrely on these spurious cues😮
Features that only weakly correlate with human preferences (r_human=−0.12) are strongly predictive for models (r_model=0.36). Points above y=x suggest that models overrely on these spurious cues😮
June 6, 2025 at 4:32 PM
Indeed, preference models can easily latch on to these subtle data artifacts!
Features that only weakly correlate with human preferences (r_human=−0.12) are strongly predictive for models (r_model=0.36). Points above y=x suggest that models overrely on these spurious cues😮
Features that only weakly correlate with human preferences (r_human=−0.12) are strongly predictive for models (r_model=0.36). Points above y=x suggest that models overrely on these spurious cues😮
Where do these biases come from?🤔Our analysis suggests they originate from training data artifacts.
For eg, humans preferred structured responses >65% of the time when the alternative wasn't structured. This gives an opportunity for models to learn these patterns as heuristics!
For eg, humans preferred structured responses >65% of the time when the alternative wasn't structured. This gives an opportunity for models to learn these patterns as heuristics!
June 6, 2025 at 4:32 PM
Where do these biases come from?🤔Our analysis suggests they originate from training data artifacts.
For eg, humans preferred structured responses >65% of the time when the alternative wasn't structured. This gives an opportunity for models to learn these patterns as heuristics!
For eg, humans preferred structured responses >65% of the time when the alternative wasn't structured. This gives an opportunity for models to learn these patterns as heuristics!
How severe is the problem? Using controlled counterfactual pairs, we found that preference models (incl. LLM evaluators) prefer biased responses in >60% of cases (defined as skew) and show high miscalibration (~40%) wrt humans.
Vagueness & sycophancy are especially problematic!
Vagueness & sycophancy are especially problematic!
June 6, 2025 at 4:32 PM
How severe is the problem? Using controlled counterfactual pairs, we found that preference models (incl. LLM evaluators) prefer biased responses in >60% of cases (defined as skew) and show high miscalibration (~40%) wrt humans.
Vagueness & sycophancy are especially problematic!
Vagueness & sycophancy are especially problematic!
Preference models act as proxies for human judgements in alignment (as reward models) & evaluation, but they can be miscalibrated.
We found that they overrely on many idiosyncratic features of AI-generated text, which can lead to reward hacking & unreliable evals. Features like:
We found that they overrely on many idiosyncratic features of AI-generated text, which can lead to reward hacking & unreliable evals. Features like:
June 6, 2025 at 4:32 PM
Preference models act as proxies for human judgements in alignment (as reward models) & evaluation, but they can be miscalibrated.
We found that they overrely on many idiosyncratic features of AI-generated text, which can lead to reward hacking & unreliable evals. Features like:
We found that they overrely on many idiosyncratic features of AI-generated text, which can lead to reward hacking & unreliable evals. Features like:
Ever wondered what makes language models generate overly verbose, vague, or sycophantic responses?
Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
June 6, 2025 at 4:32 PM
Ever wondered what makes language models generate overly verbose, vague, or sycophantic responses?
Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
Our new paper investigates these and other idiosyncratic biases in preference models, and presents a simple post-training recipe to mitigate them! Thread below 🧵↓
Reposted by Chaitanya Malaviya
Evaluating language model responses on open-ended tasks is hard! 🤔
We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️.
EvalAgent identifies 👩🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇
We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️.
EvalAgent identifies 👩🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇
April 22, 2025 at 3:04 PM
Evaluating language model responses on open-ended tasks is hard! 🤔
We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️.
EvalAgent identifies 👩🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇
We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️.
EvalAgent identifies 👩🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇
Joint work done @ai2.bsky.social with Joseph Chee Chang, Dan Roth, Mohit Iyyer, Mark Yatskar, @kylelo.bsky.social .
Find these & many more results in our paper: arxiv.org/abs/2411.07237
Use our code: github.com/allenai/Cont...
Explore our data: huggingface.co/datasets/all...
Find these & many more results in our paper: arxiv.org/abs/2411.07237
Use our code: github.com/allenai/Cont...
Explore our data: huggingface.co/datasets/all...
Contextualized Evaluations: Taking the Guesswork Out of Language Model Evaluations
Language model users often issue queries that lack specification, where the context under which a query was issued -- such as the user's identity, the query's intent, and the criteria for a response t...
arxiv.org
November 13, 2024 at 2:16 PM
Joint work done @ai2.bsky.social with Joseph Chee Chang, Dan Roth, Mohit Iyyer, Mark Yatskar, @kylelo.bsky.social .
Find these & many more results in our paper: arxiv.org/abs/2411.07237
Use our code: github.com/allenai/Cont...
Explore our data: huggingface.co/datasets/all...
Find these & many more results in our paper: arxiv.org/abs/2411.07237
Use our code: github.com/allenai/Cont...
Explore our data: huggingface.co/datasets/all...
🤔 How can we use context to learn more about model behavior?
We can study "default" responses from models. Under what type of context does their response get highest score?
We uncover a bias towards WEIRD contexts (Western, Educated, Industrialized, Rich & Democratic)!
We can study "default" responses from models. Under what type of context does their response get highest score?
We uncover a bias towards WEIRD contexts (Western, Educated, Industrialized, Rich & Democratic)!
November 13, 2024 at 2:16 PM
🤔 How can we use context to learn more about model behavior?
We can study "default" responses from models. Under what type of context does their response get highest score?
We uncover a bias towards WEIRD contexts (Western, Educated, Industrialized, Rich & Democratic)!
We can study "default" responses from models. Under what type of context does their response get highest score?
We uncover a bias towards WEIRD contexts (Western, Educated, Industrialized, Rich & Democratic)!
🤔 Does providing context to evaluators have a substantial effect on evaluation conclusions?
We find that (1) presence of context can improve agreement between evaluators and (2) even change model rankings! 🤯
We find that (1) presence of context can improve agreement between evaluators and (2) even change model rankings! 🤯
November 13, 2024 at 2:16 PM
🤔 Does providing context to evaluators have a substantial effect on evaluation conclusions?
We find that (1) presence of context can improve agreement between evaluators and (2) even change model rankings! 🤯
We find that (1) presence of context can improve agreement between evaluators and (2) even change model rankings! 🤯
...we then conduct experiments providing context (1) during response generation, (2) during evaluation or (3) both.
November 13, 2024 at 2:16 PM
...we then conduct experiments providing context (1) during response generation, (2) during evaluation or (3) both.
With ✨Contextualized Evaluations✨, we synthetically generate context as clarifying, follow-up questions to an underspecified query...
November 13, 2024 at 2:16 PM
With ✨Contextualized Evaluations✨, we synthetically generate context as clarifying, follow-up questions to an underspecified query...
Underspecified queries can lead to arbitrary evaluation judgments of response quality!
e.g., Given a query “Is coffee good for you?”, how can evaluators accurately judge model responses when they aren't informed about the user’s preferences, background or important criteria?
e.g., Given a query “Is coffee good for you?”, how can evaluators accurately judge model responses when they aren't informed about the user’s preferences, background or important criteria?
November 13, 2024 at 2:16 PM
Underspecified queries can lead to arbitrary evaluation judgments of response quality!
e.g., Given a query “Is coffee good for you?”, how can evaluators accurately judge model responses when they aren't informed about the user’s preferences, background or important criteria?
e.g., Given a query “Is coffee good for you?”, how can evaluators accurately judge model responses when they aren't informed about the user’s preferences, background or important criteria?
Underspecified queries are prevalent in many datasets used to benchmark language models (e.g., Chatbot Arena, AlpacaEval).
These can be ambiguous (e.g., what is a transformer? ... 🤔 for NLP or EE?), subjective (e.g., who is the best? ... 🤔 what criteria?), and more!
These can be ambiguous (e.g., what is a transformer? ... 🤔 for NLP or EE?), subjective (e.g., who is the best? ... 🤔 what criteria?), and more!
November 13, 2024 at 2:16 PM
Underspecified queries are prevalent in many datasets used to benchmark language models (e.g., Chatbot Arena, AlpacaEval).
These can be ambiguous (e.g., what is a transformer? ... 🤔 for NLP or EE?), subjective (e.g., who is the best? ... 🤔 what criteria?), and more!
These can be ambiguous (e.g., what is a transformer? ... 🤔 for NLP or EE?), subjective (e.g., who is the best? ... 🤔 what criteria?), and more!
Excited to share ✨ Contextualized Evaluations ✨!
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
November 13, 2024 at 2:16 PM
Excited to share ✨ Contextualized Evaluations ✨!
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓
Benchmarks like Chatbot Arena contain underspecified queries, which can lead to arbitrary eval judgments. What happens if we provide evaluators with context (e.g who's the user, what's their intent) when judging LM outputs? 🧵↓