Lightnews — Scholar-powered news

Danica Dillion @danicajdillion.bsky.social · 19d

We hope WorldValuesBench supports building LLMs that are safer, fairer, and more culturally aware.

Led by Wenlong Zhao with Debanjan Mondal, Niket Tandon,
@kurtjgray.bsky.social, and Yuling Gu

Code: github.com/Demon702/Wor...

Paper: aclanthology.org/2024.lrec-ma...

WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models

Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, Yuling Gu. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluatio...

aclanthology.org

1

Danica Dillion @danicajdillion.bsky.social · 19d

The World Values Survey is one of the most widely used datasets on global values:
🌏 94,728 participants across 64 countries in the latest wave
📊 Representative national samples
🗣️ Primarily face-to-face interviews
❓ 200+ value questions spanning social, political, economic, and moral domains

1

Danica Dillion @danicajdillion.bsky.social · 19d

What makes WorldValuesBench different?
🎯 Fine-grained demographics: 42 attributes (age, gender, country, urban/rural, etc.)
🌐 Breadth of values: social norms, trust, economics, religion, politics, and more
📊 Scale: 20M examples for evaluation
📏 Distributions rather than averages of human values

1 1

Danica Dillion @danicajdillion.bsky.social · 19d

LLMs are already shaping advice, education, and policy around the world. To be safe and serve global users, it's important they reflect cultural variation in values rather than a single “global average” or disproportionately Western views.

1

Danica Dillion @danicajdillion.bsky.social · 19d

🌍 Introducing WorldValuesBench!

A benchmark to evaluate how well LLMs reflect cultural differences in human values.

Built from 94k+ participants in the World Values Survey → 20M examples of (demographics, value question → answer).

🧵

1 1

Danica Dillion @danicajdillion.bsky.social · Aug 26

Huge thanks to brilliant coauthors @helenldevine.bsky.social and @kurtjgray.bsky.social!

Preprint available here: doi.org/10.31234/osf...

Data, code, and preregs here: osf.io/d94b8/

OSF

doi.org

2

Danica Dillion @danicajdillion.bsky.social · Aug 26

Dogs are amazing companions.

But when love for dogs surpasses love for people, it can come at a cost: less concern for others and deeper social disconnection.

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

For many, dogs aren’t just valued as much as people—they’re valued more.

This can have real impacts: Owners chose to give more to animal charities than those helping people, including a children's hospital and food bank. 🐶>🏥🥫

Even children in need can come second to dogs.

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

Dog owners were more likely than non-owners to prioritize dogs, but even non-owners scored high on these measures.

This suggests that seeing dogs as soulmates is widespread. 🐶💞

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

The same trend appears locally:

US counties with lower birth rates have more pet stores and higher pet industry earnings, even after controlling for population and GDP.

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

As the US birth rate falls, pet spending rises—with a strong negative correlation (r = -.93)

This link holds even after accounting for inflation, GDP, population, the poverty rate, and median age.

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

Dog-over-human choices were strongest among owners without kids. Eg:

🐶 73% of childless owners would save their dog over a stranger
🍼 Only 44% of owners with kids would do the same

Childless owners were also more likely to say their dog takes the place of a baby.

And looking @ national trends...

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

56% of owners would save their own dog over a person. And it doesn't stop there:

🐶 25% would give $50 to a puppy over a child in need

🐶 43% would feed a hungry puppy over a hungry stranger

🐶 20% would save an unfamiliar puppy over a human life

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

We wanted to know whether this soulmate bond shifts moral concern from people to dogs.

Would owners pick a dog’s life over a human’s—even a stranger’s dog?

So we posed moral dilemmas pitting dogs against people 💁⚖️🐶

1 1

Danica Dillion @danicajdillion.bsky.social · Aug 26

It’s no surprise that people who see their dog as a soulmate pamper them—sharing plates, beds, and fancy treats.

But many go further: preferring dogs over friends, and choosing dog time over people time.

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

73% of dog owners see their dog as a soulmate ❤️🐶

They endorse statements like:
"My dog is my main companion"
"My dog is my main emotional support"
"My dog’s love is purer than most people’s love"

1

Danica Dillion @danicajdillion.bsky.social · Aug 26

1 in 5 dog owners would save a puppy's life over yours. 🐶

The majority of pet owners see their dog as a soulmate. Many don’t just prefer their own dog over people—they’ll pick a stranger’s dog over a human life.

New preprint 🧵

1 2

Danica Dillion @danicajdillion.bsky.social · Aug 25

Many thanks to my amazing co-authors Debanjan Mondal, Wenlong Zhao, Niket Tandon, and @kurtjgray.bsky.social!

Preprint link: doi.org/10.31234/osf...

OSF

doi.org

Danica Dillion @danicajdillion.bsky.social · Aug 25

AI is already shaping moral decisions in high-stakes domains like the military, healthcare, and education.

We need AI that reflects people’s values and is more transparent in its decisions.

Our work shows how moral psychology can help guide the design of more interpretable, morally aligned AI.

1

Danica Dillion @danicajdillion.bsky.social · Aug 25

It’s also promising that bottlenecks beat the baseline across several moral frameworks.

For practical uses, bottlenecks should be tailored to the people they serve and the models they’re used with.

We’re hopeful similar approaches can be adapted to different use cases.

1

Danica Dillion @danicajdillion.bsky.social · Aug 25

One great thing about the bottleneck method is that it’s simple. It doesn’t require additional training or data.

1

Danica Dillion @danicajdillion.bsky.social · Aug 25

How do people feel about the bottleneck method?

In a preregistered study (n=239), an AI described as using a cognitive bottleneck was rated as more transparent, trustworthy, and moral than a baseline end-to-end model.

1

Danica Dillion @danicajdillion.bsky.social · Aug 25

Of course, bottlenecks can’t make LLMs fully transparent, as their inner workings are too complex to trace.

But bottleneck feature ratings explained over 90% of the variance in model moral scores, adding a layer of interpretability.

1

Danica Dillion @danicajdillion.bsky.social · Aug 25

We set out to make AI more transparent without losing accuracy. Instead, it got *better* at matching people’s moral judgments.

Bottlenecks beat the baseline in nearly every moral framework × LLM pairing.

1

Danica Dillion @danicajdillion.bsky.social · Aug 25

We tested bottleneck models for moral judgment across 6 moral frameworks × 6 LLMs (36 analyses)

Baseline: LLM reads a scenario → outputs a moral score

Bottleneck: LLM reads a scenario → rates features (e.g., harm, intent) → moral score derived from these ratings

1