Thrilled to share our research showing how LLM models can be influenced by bias from "spun" medical literature is now featured in Northeastern's Khoury news! This shows critical insights as AI enters healthcare.
The full paper can be found at arxiv.org/abs/2502.07963

I am at CHIL this week to present my poster (Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?) on Thursday June 26.

Looking forward to connecting and sharing our work on spin with the CHIL community!

I am at CHI this week to present my poster (Framing Health Information: The Impact of Search Methods and Source Types on User Trust and Satisfaction in the Age of LLMs) on Wednesday April 30

CHI Program Link: programs.sigchi.org/chi/2025/pro...

Looking forward to connecting with you all!

LLM-based chatbots are changing how people search for health information—but how do users perceive their quality and trustworthiness compared to other online sources? Our survey study explores these questions. Check it out! www.jmir.org/2025/1/e68560

I'm searching for some comp/ling experts to provide a precise definition of “slop” as it refers to text (see: corp.oup.com/word-of-the-...)

I put together a google form that should take no longer than 10 minutes to complete: forms.gle/oWxsCScW3dJU...
If you can help, I'd appreciate your input! 🙏

Huge thanks to my amazing co-authors!
Karen Y.C. Zhang, @ramezkouzy.bsky.social, @ijmarshall.bsky.social, @jessyjli.bsky.social, & @byron.bsky.social [7/7]

Check out our full findings here: arxiv.org/abs/2502.07963

Can we fix this? We tested zero-shot prompts to reduce LLMs' susceptibility to spin.
Good news: prompts that encouraged reasoning reduced their tendency to overstate trial results! 🛠️
Careful design is key to improving evidence synthesis for clinical decisions. [6/7]

When we asked LLMs to simplify abstracts into plain language, they often propagated spin into their summaries. This means LLMs could unintentionally mislead patients and non-experts about the effectiveness of treatments. 😱 [5/7]

We asked LLMs how favorably they perceived a treatment’s results (0-10 scale). Even though LLMs could detect spin, they were far more influenced by it than human experts.
Meaning: LLMs believed spun abstracts presented more favorable results! 😬 [4/7]

When we prompted 22 LLMs to identify spin in medical abstracts, we found that they were moderately to strongly capable of detecting spin.
However, things got interesting when we asked LLMs to interpret the results… [3/7]
🔽

So what is spin?
Spin refers to reporting strategies that make experimental treatments appear more beneficial than they actually are—often distracting from nonsignificant results.

Example:
❌ “The treatment shows a promising trend toward significance…”
✅ “No significant difference was found.”
[2/7]

🚨 Do LLMs fall for spin in medical literature? 🤔

In our new preprint, we find that LLMs are susceptible to biased reporting of clinical treatment benefits in abstracts—more so than human experts. 📄🔍 [1/7]

Full Paper: arxiv.org/abs/2502.07963

🧵👇

As someone interested in an academic position post-PhD, I found this post very helpful. Thank you for sharing your wisdom and advice.

Awesome! Thank you

Thank you!

The application form says it is no longer accepting responses. Is the application closed now?