Bufan Gao
@bufangao.bsky.social
Psychology PhD student @UChicago
π jouisseuse.github.io
π jouisseuse.github.io
Excited to present at #EMNLP2025
Really appreciate @elisakreiss.bsky.socialβs kind guidance and encouragement throughout this work π
Really appreciate @elisakreiss.bsky.socialβs kind guidance and encouragement throughout this work π
September 11, 2025 at 4:01 PM
Excited to present at #EMNLP2025
Really appreciate @elisakreiss.bsky.socialβs kind guidance and encouragement throughout this work π
Really appreciate @elisakreiss.bsky.socialβs kind guidance and encouragement throughout this work π
π Our results highlight the brittleness of current bias evaluations: small prompt changes can reverse conclusions.
π Paper: arxiv.org/abs/2509.04373
π» Code: github.com/jouisseuse/B...
π Paper: arxiv.org/abs/2509.04373
π» Code: github.com/jouisseuse/B...
Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases
As LLMs are increasingly applied in socially impactful settings, concerns about gender bias have prompted growing efforts both to measure and mitigate such bias. These efforts often rely on evaluation...
arxiv.org
September 11, 2025 at 4:01 PM
π Our results highlight the brittleness of current bias evaluations: small prompt changes can reverse conclusions.
π Paper: arxiv.org/abs/2509.04373
π» Code: github.com/jouisseuse/B...
π Paper: arxiv.org/abs/2509.04373
π» Code: github.com/jouisseuse/B...
When prompts contain cues typical of gender bias evaluation setups, models shift pronoun use: fewer βhe,β more βthey.β
This suggests that LLM benchmark behavior may generalize less and less to non-benchmark settings, raising new concerns about ecological validity.
This suggests that LLM benchmark behavior may generalize less and less to non-benchmark settings, raising new concerns about ecological validity.
September 11, 2025 at 4:01 PM
When prompts contain cues typical of gender bias evaluation setups, models shift pronoun use: fewer βhe,β more βthey.β
This suggests that LLM benchmark behavior may generalize less and less to non-benchmark settings, raising new concerns about ecological validity.
This suggests that LLM benchmark behavior may generalize less and less to non-benchmark settings, raising new concerns about ecological validity.
π¨ New #EMNLP2025 paper!
Do LLMs exhibit distinct behavior when the prompt looks similar to common evaluation prompts? π
We show that prompts that signal bias evaluation can flip the measured bias. See below β¬οΈ
Do LLMs exhibit distinct behavior when the prompt looks similar to common evaluation prompts? π
We show that prompts that signal bias evaluation can flip the measured bias. See below β¬οΈ
September 11, 2025 at 4:01 PM
π¨ New #EMNLP2025 paper!
Do LLMs exhibit distinct behavior when the prompt looks similar to common evaluation prompts? π
We show that prompts that signal bias evaluation can flip the measured bias. See below β¬οΈ
Do LLMs exhibit distinct behavior when the prompt looks similar to common evaluation prompts? π
We show that prompts that signal bias evaluation can flip the measured bias. See below β¬οΈ