Lightnews — Scholar-powered news

Reposted by Hope Schroeder

Nancy Baym @nancybaym.bsky.social · Jul 28

We may have the chance to hire an outstanding researcher 3+ years post PhD to join Tarleton Gillespie, Mary Gray and me in Cambridge MA bringing critical sociotechnical perspectives to bear on new technologies.

jobs.careers.microsoft.com/global/en/jo...

Search Jobs | Microsoft Careers

https://jobs.careers.microsoft.com/global/en/job/1849026/Principal-Researcher-–-Sociotechnical-Systems-–-Microsoft-Research

49 89

Hope Schroeder @hopeschroeder.bsky.social · Jul 24

Thanks for sharing- not just our paper but also learned a lot from this list! :)

1

Hope Schroeder @hopeschroeder.bsky.social · Jul 23

Awesome work and great presentation! Congrats!! ⚡️

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

Talking about this work tomorrow (Wed, July 23rd) at #IC2S2 in Norrköping during the 11 am session on LLMs, Annotation, and Synthetic Data! Come hear about this and more!

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

🗣️ Excited to share our new #ACL2025 Findings paper: “Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks” with Jad Kabbara and Deb Roy. Arxiv: arxiv.org/abs/2507.15821
Read about our findings ⤵️

Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks

LLM use in annotation is becoming widespread, and given LLMs' overall promising performance and speed, simply "reviewing" LLM annotations in interpretive tasks can be tempting. In subjective annotatio...

arxiv.org

2 14

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

👋 ☺️

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

Implications vary by task and domain. Researchers should clearly define their annotation constructs before reviewing LLM annotations. We are subject to anchoring bias that can affect our evaluations, or even our research findings!
Read more: arxiv.org/abs/2507.15821

5

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

Using LLM-influenced labels, even when a crowd of humans reviews them and is aggregated into a set of crowd labels, can lead to 1) different findings when used in data analysis and 2) different results when used as a basis of evaluating LLM performance on the task.

1 2

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

What happens if we use LLM-influenced labels as ground truth when evaluating LLM performance on these tasks? We can seriously overestimate LLM performance on these tasks. F1 scores for some tasks were +.5 higher when evaluated using LLM-influenced labels as ground truth!

1 3

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

However… annotators STRONGLY took LLM suggestions: just 40% of human crowd labels overlap with LLM baselines, but overlap jumps to over 80% when LLM suggestions are given (varied crowd thresholds and conditions shown in graph). Beware: Humans are subject to anchoring bias!

1 1 6

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

Some findings: ⚠️ reviewing LLM suggestions did not make annotators go faster, and often slowed them down! OTOH, having LLM assistance made annotators ❗more self-confident❗in their task and content understanding at no identified cost to their tested task understanding.

1 2 5

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

We conducted experiments where over 410 unique annotators generated with over 7,000 annotations across three LLM assistance conditions of varying strengths against a control, using two different models, and two different complex, subjective annotation tasks.

1 1 3

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

LLMs can be fast and promising annotators, so letting human annotators "review" first-pass LLM annotations in interpretive tasks is tempting. How does this impact productivity, annotators, evaluating LLM performance on subjective tasks and downstream data analysis?

1 2

Hope Schroeder @hopeschroeder.bsky.social · Jul 22

🗣️ Excited to share our new #ACL2025 Findings paper: “Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks” with Jad Kabbara and Deb Roy. Arxiv: arxiv.org/abs/2507.15821
Read about our findings ⤵️

Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks

LLM use in annotation is becoming widespread, and given LLMs' overall promising performance and speed, simply "reviewing" LLM annotations in interpretive tasks can be tempting. In subjective annotatio...

arxiv.org

4 8 40

Hope Schroeder @hopeschroeder.bsky.social · Jun 25

Thanks for attending and for your comments!!

1

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

Come hear much more at our paper talk tomorrow (Wednesday, 6/25) in Meta Research and Critiques, 9:12 am at New Stage A! Read the paper here: dl.acm.org/doi/10.1145/...

Disclosure without Engagement: An Empirical Review of Positionality Statements at FAccT | Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency

You will be notified whenever a record that you have chosen has been cited.

dl.acm.org

1 4

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

*What should FAccT do?* We discuss a need for the conference to clarify its policies next year, engage scholars from different disciplines when considering policy on this delicate subject, and engage authors in reflexive practice upstream of paper-writing, potentially through CRAFT.

1 1

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

*Are disclosed features connected to described impacts?* Disclosed features are much less commonly described in terms of impacts the feature had on the research, which may leave a gap for readers to jump to conclusions about how the disclosed feature impacted the work.

1 1 3

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

*What do authors disclose in positionality statements?* We conducted fine-grained annotation of the statements. We find academic background and training are disclosed most often, but identity features like race and gender are also common.

1 2

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

We reviewed papers from the entire history of FAccT for the presence of positionality statements. We find 2024 marked a significant proportional increase in papers that included positionality statements, likely as a result of PC recommendations:

1

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

In 2024, the FAccT PCs recommended that authors write positionality statements (blog post: medium.com/@alexandra.o...)

Responsible AI Research Needs Impact Statements Too

Alexandra Olteanu, Michael Ekstrand, Carlos Castillo, and Jina Suh

medium.com

1 1

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

With ongoing reflection on the impact of computing on society, and the role of researchers in shaping impacts, positionality statements have become more common in computing venues, but little is known about their contents or the impact of conference policy on their presence.

1 1

Hope Schroeder @hopeschroeder.bsky.social · Jun 24

1) Thrilled to be at #FAccT for the first time this week, representing a meta-research paper on positionality statements at FAccT from 2018-2024, in collaboration with @s010n.bsky.social and Akshansh Pareek, "Disclosure without Engagement: An Empirical Review of Positionality Statements at FAccT"

1 1 10

Hope Schroeder @hopeschroeder.bsky.social · Jun 6

Democracy needs you! Super excited to be co-organizing this NLP for Democracy workshop at #COLM2025 with a wonderful group. Abstracts due June 19th!

Julia Mendelsohn @jmendelsohn2.bsky.social · May 21

📣 Super excited to organize the first workshop on ✨NLP for Democracy✨ at COLM @colmweb.org!!

Check out our website: sites.google.com/andrew.cmu.e...

Call for submissions (extended abstracts) due June 19, 11:59pm AoE

#COLM2025 #LLMs #NLP #NLProc #ComputationalSocialScience

NLP 4 Democracy - COLM 2025

sites.google.com

2 8

Reposted by Hope Schroeder

Hanna Wallach @hannawallach.bsky.social · May 20

Exciting news: The Fairness, Accountability, Transparency and Ethics (FATE) group at Microsoft Research NYC is hiring a predoctoral fellow!!! 🎉

www.microsoft.com/en-us/resear...

FATE Research Assistant (“Pre-doc”) - Microsoft Research

The Fairness, Accountability, Transparency, and Ethics (FATE) Research group at Microsoft Research New York City (MSR NYC) is looking for a pre-doctoral research assistant (pre-doc) to start August 20...

www.microsoft.com

2 13 33

Reposted by Hope Schroeder

Artjoms Šeļa @artjomshl.bsky.social · May 7

a whole cluster of postdocs and phd positions in Tartu in Digital Humanities / Computational Social Science / AI under the umbrella of big European projects.

consider sharing please!

Andres Karjus @andreskarjus.bsky.social · May 5

While NSF freezes funding and UK leading unis cut departments, EU still supports important research. 2 Horizon funded centres in #Tartu #Estonia hiring phds&postdocs right now in CSS, AI, DH, text corpora. See
- post below
- DigiTS ut.ee/en/job-offer... (our good colleagues in humanities).

42 35