Lightnews — Scholar-powered news

Reposted by Elinor🎗️ @ COLM 🍁

David Mimno @dmimno.bsky.social · 1d

COLM word cloud. Yoav says it’s the year of reasoning, but evaluation is also huge.

Evaluation reasoning interpretability rl in context benchmark alignment synthetic data

3 5 30

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 2d

bsky.app/profile/elin...

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

🚨 New preprint! 🚨
Excited to share my work: An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies 🤖🗳️

I’ll be presenting this at @colmweb.org in the NLP4Democracy workshop!

🔗 arxiv.org/abs/2509.12577

An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies

In an era of increasing societal fragmentation, political polarization, and erosion of public trust in institutions, representative deliberative assemblies are emerging as a promising democratic forum...

arxiv.org

1

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 2d

I’m at #COLM2025! Would love to chat about anything related to pluralistic alignment, fairness evaluations, societal impacts of LLMs, etc 😊

You can also find me at the NLP4Democracy workshop giving a talk about my work analyzing democratic deliberation with LLMs Oct 10th!

1 2

Reposted by Elinor🎗️ @ COLM 🍁

No Score Draws ✍🏻⚽ @noscoredraws.com · 2d

Alright the evening sky, you’re utterly wondrous and fantastical, we get it, geez

4 11 220

Reposted by Elinor🎗️ @ COLM 🍁

Maria Antoniak @mariaa.bsky.social · 2d

Here’s a #COLM2025 feed!

Pin it 📌 to follow along with the conference this week!

2 17 24

Reposted by Elinor🎗️ @ COLM 🍁

Julia Mendelsohn @jmendelsohn2.bsky.social · 2d

I will be at #COLM2025 this week, and would love to connect with folks interested in applications (and critiques) of language modeling in social science research!

And join us for the NLP4Democracy workshop on Friday!

sites.google.com/andrew.cmu.e...

#NLP #NLProc #LLM #ComputationalSocialScience

NLP 4 Democracy - COLM 2025

sites.google.com

5 16

Reposted by Elinor🎗️ @ COLM 🍁

Andy Halterman @ahalterman.bsky.social · 19d

Very excited that my paper with @katakeith.bsky.social is now out in @polanalysis.bsky.social. We investigate whether LLMs actually follow the instructions/definitions provided in codebooks, propose some diagnostics, and release a new evaluation dataset.
www.cambridge.org/core/journal...

Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts | Political Analysis | Cambridge Core

Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts

www.cambridge.org

14 31

Reposted by Elinor🎗️ @ COLM 🍁

Naomi Saphra @nsaphra.bsky.social · 19d

I wish students understood in most empirical AI research there’s a huge scientific advantage from being constitutionally excited by math vs intimidated, but very little additional gain from being actually “good” at math. Maybe they’d be less intimidated if they didn’t feel they had to be “good”.

7 4 51

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

This work is part of my master’s thesis at @mit.edu @medialab.bsky.social, supervised by Deb Roy and with the help of Jad Kabbara @jad-kabbara.bsky.social.

🔗 arxiv.org/abs/2509.12577

An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies

In an era of increasing societal fragmentation, political polarization, and erosion of public trust in institutions, representative deliberative assemblies are emerging as a promising democratic forum...

arxiv.org

1

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

Beyond research, this paves the way for:
✨ Tools supporting live assemblies in real time
✨ Increasing transparency & communicating critical insights to decision-makers
✨ Enabling richer cross-assembly analysis to advance research on deliberative best practices

1

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

In the tech-enhanced assembly, our framework revealed:
🔹 How deliberation surfaced, refined, or discarded ideas
🔹 *Missing* viable ideas
🔹 How opinion shifts & rec edits shaped outcomes
🔹 Underlying values & trade-offs invisible to decision-makers

1

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

We develop an LLM-based framework to:
✅ Map how suggestions transform into concrete recommendations
✅ Reconstruct individuals’ evolving perspectives
✅ Detect why votes shift across deliberation

1

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

Despite their promise, we still lack tools to empirically trace:
• how ideas evolve into recommendations
• how deliberation shapes perspectives & votes

At MIT CCC, we hosted our own tech-enhanced assembly to explore how AI can help!

sustainabilityassembly.portal.cortico.ai

Loading...

sustainabilityassembly.portal.cortico.ai

1

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

Deliberative assemblies bring together everyday citizens selected by lottery. Through deliberation 💬 & learning, they collectively form policy recommendations 💡for decision-makers.

They’ve proven successful worldwide, facilitating rebuilding trust & strengthening democracy 🤝.

1 1

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 21d

🚨 New preprint! 🚨
Excited to share my work: An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies 🤖🗳️

I’ll be presenting this at @colmweb.org in the NLP4Democracy workshop!

🔗 arxiv.org/abs/2509.12577

An AI-Powered Framework for Analyzing Collective Idea Evolution in Deliberative Assemblies

In an era of increasing societal fragmentation, political polarization, and erosion of public trust in institutions, representative deliberative assemblies are emerging as a promising democratic forum...

arxiv.org

1 1 6

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 23d

"This suggests that LLM benchmark behavior may generalize less and less to non-benchmark settings, raising new concerns about ecological validity."

super interesting

Bufan Gao @bufangao.bsky.social · 27d

🚨 New #EMNLP2025 paper!

Do LLMs exhibit distinct behavior when the prompt looks similar to common evaluation prompts? 👀

We show that prompts that signal bias evaluation can flip the measured bias. See below ⬇️

Violin plots of Probability of Pronoun Shift. Models show significant sensitivity to prompt changes: when prompts highlight gender evaluation, pronoun use shifts, with decreased “he” and increased “they” use.

1

Reposted by Elinor🎗️ @ COLM 🍁

Yoshua Bengio @yoshuabengio.bsky.social · 23d

This paper yields the same conclusion as what @mustafasuleymanai.bsky.social recently posted on the danger of 'seemingly conscious AI'.

mustafa-suleyman.ai/seemingly-co...

We must build AI for people; not to be a person

mustafa-suleyman.ai

4 15

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 24d

Thread inspired by having to review 6(!!) papers for AAAI and most of them having no line numbers. And one particularly great paper I want to show the authors exactly how much I enjoyed it via my annotation drawings (>20 check marks, ~10 exclamations, and even 2 hearts!)

3

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 24d

Typing the (sometimes extreme) number of typo corrections is tedious, time consuming, and especially frustrating when there’s no line numbers on the pdf! It would honestly be faster for me to edit it myself 🙄

1 2

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 24d

When reading papers, especially reviewing, I like to print and annotate as I read. I wish I could upload this to open review so authors can see smaller suggestions (typos, formatting errors) as well as smaller positive notes eg things I appreciated or found useful/interesting

1 4

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 24d

!!!!! me too

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 24d

i'm curious to hear your thoughts on their piloting of AI assisted reviewing: bsky.app/profile/aaai...

The Association for the Advancement of Artificial Intelligence @aaai.org.web.brid.gy · May 16

AAAI Launches AI-Powered Peer Review Assessment System

AAAI today announced a pilot program that strategically incorporates Large Language Models (LLMs) to enhance the academic paper review process for the AAAI-26 conference. The post AAAI Launches AI-Powered Peer Review Assessment System appeared first on AAAI.

aaai.org

1 3

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 24d

totally agree! the peer review system is already over burdened and there needs to be an intermediate step for AI generated work. for example, AAAI received *double* the amount of normal submissions this year even after desk rejections aaai.org/conference/a....

AAAI-26 Review Process Update: Scale, Integrity Measures, and Pathways to Sustainability - AAAI

aaai.org

1 1 8

Elinor🎗️ @ COLM 🍁 @elinorpd.bsky.social · 25d

First crochet project done! Super proud and excited for the next one

7

Reposted by Elinor🎗️ @ COLM 🍁

Ben Recht @beenwrekt.bsky.social · 27d

This new center strikes the right tone in approaching the AI alignment problem. alignmentalignment.ai

Center for the Alignment of AI Alignment Centers

We align the aligners

alignmentalignment.ai

4 14 58