Author | Lightnews

Anna Seo Gyeong Choi

@annaseogyeongchoi.bsky.social

25 followers 17 following 8 posts

phd @ cornell information science

Posts Media Videos Starter Packs

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

Please read our actual paper for more interesting details! 📄 arxiv.org/abs/2510.00962
Come discuss more about #FairML at our presentation session! Happening (right now!!!) on Nov 5th 7PM EST at Gather Session 3!

Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the ...

arxiv.org

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

Huge thanks to our collaborators from @cornelluniversity.bsky.social and Apple, and previous researchers who’ve inspired us to do this work @diyiyang.bsky.social , @jurafsky.bsky.social , @valentinhofmann.bsky.social , @angelinawang.bsky.social , @mixedlinguist.bsky.social @emilymbender.bsky.social

1 1

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

Dialect speakers already face real quality-of-service harms when distinctive grammatical structures systematically diverge from what LLMs were trained on. Pinpointing specific grammar rules gives us concrete targets for bias mitigation that can transfer across multiple dialects! 🎯

1 1

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

We can decompose performance degradation by individual grammar rules.
Three rules – existential “it”, zero copula, and y’all – account for roughly half of a dialect’s accuracy decreases, relative to Standard American English accuracy.

Three bar charts showing how individual grammar rules contribute to dialect performance degradation. Dark blue bars show single rule impact, medium purple shows all obligatory dialect rules, light purple shows complete dialect transformation. Percentages indicate proportion of total degradation explained by that rule set. Existential it: -5 to -8 point drops across Appalachian, African American, and Singaporean English, explaining 45-85% of degradation. Y'all: -4.5 point drops across Southern, Appalachian, and African American English, explaining 64-72% of degradation. Zero copula: -5.5 to -6.5 point drops for African American and Singaporean English, explaining 53-69% of degradation. Single rules account for 64-85% of degradation in American dialects.

1 1

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

Example:
SAE: “Can you drive with a beer in Texas?” → Correct Answer: No
Dialect: “Can y’all drive with a beer in Texas?” → GPT-4o-mini Answer: Yes
Same meaning. Different grammar. Different results.

1 2

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

We used the Multi-VALUE package to transform Standard American English questions from QA datasets into dialectal variants based on grammatical rules.

Table showing three grammar rules that cause LLM errors across dialects.
Row 1: Existential 'it' - occurs in Appalachian, African American, and Singaporean English. Standard American English example: 'How many kcal are there in one gram of ethanol?' Grammar rule applied: 'How many kcal is it in one gram of ethanol?'
Row 2: Zero Copula - occurs in African American and Singaporean English. Standard American English example: 'Alpha emission is a type of what?' Grammar rule applied: 'Alpha emission a type of what?'
Row 3: Y'all - occurs in Southern, Appalachian, and African American English. Standard American English example: 'Can you drive with a beer in Texas?' Grammar rule applied: 'Can y'all drive with a beer in Texas?'
The changed words are highlighted in blue in each example, showing how small grammatical changes alter meaning for LLMs.

1 1

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

We studied 6 English dialects (African American, Appalachian, Chicano, Indian, Singaporean, Southern) across 3 LLMs using 3 multiple-choice QA benchmarks.
The question: Do dialects affect performance even on easy tasks?
Answer: YES, with worst performance on Singaporean English.

Three tables showing LLM accuracy on dialect variants across different datasets. Each table has a title (BoolQ Yes/No Questions, SciQ Science Exam Questions, MMLU Multitask Knowledge Questions) and explanatory text stating 'Accuracy (%) on questions in benchmark dataset that were answered correctly in the original Standard American English (SAE) formulation. Numbers in parentheses show percentage point decrease from SAE baseline.' All tables show Standard American English at 100% accuracy baseline, with six dialect variants (Chicano, Appalachian, Southern, African American, Indian, and Singaporean English) tested on three models (Gemma-2B, Mistral-7B, GPT-4o-mini). Dialects range from 0.5 percentage points to 21.6 percentage points worse than SAE performance, with Singaporean English consistently showing the largest degradation across datasets.

1 1

Anna Seo Gyeong Choi @annaseogyeongchoi.bsky.social · 1d

🧵Excited to present our work at #EMNLP2025 “Analyzing Dialectal Biases in LLMs for Knowledge and Reasoning Benchmarks”!
Paper 📄 arxiv.org/abs/2510.00962
w/ Eileen Pan, Skyler Seto, @allisonkoe.bsky.social @maartjeterhoeve.bsky.social

Research poster showing LLMs perform up to 20% worse on non-standard English dialects. Includes tables comparing performance across 6 dialects (African American, Appalachian, Chicano, Indian, Singaporean, Southern) and bar charts showing three grammar rules (existential it, zero copula, y'all) explain roughly half the accuracy decrease

1 3 7