Lightnews — Scholar-powered news

Claire Gillan @clairegillan.bsky.social · 13d

Looks interesting! We have been facing this exact issue - finding big inconsistencies across different LLMs rating the same text.

Joachim Baumann @joachimbaumann.bsky.social · 27d

🚨 New paper alert 🚨 Using LLMs as data annotators, you can produce any scientific result you want. We call this **LLM Hacking**.

Paper: arxiv.org/pdf/2509.08825

$We present our new preprint titled "Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation". We quantify LLM hacking risk through systematic replication of 37 diverse computational social science annotation tasks. For these tasks, we use a combined set of 2,361 realistic hypotheses that researchers might test using these annotations. Then, we collect 13 million LLM annotations across plausible LLM configurations. These annotations feed into 1.4 million regressions testing the hypotheses. For a hypothesis with no true effect (ground truth $p > 0.05$), different LLM configurations yield conflicting conclusions. Checkmarks indicate correct statistical conclusions matching ground truth; crosses indicate LLM hacking -- incorrect conclusions due to annotation errors. Across all experiments, LLM hacking occurs in 31-50\% of cases even with highly capable models. Since minor configuration changes can flip scientific conclusions, from correct to incorrect, LLM hacking can be exploited to present anything as statistically significant.$

5 4

Reposted by Claire Gillan

Sarah Katharina Buehler @sarahkathbuehler.bsky.social · 20d

Excited to share this new preprint on the Predictive Modelling of Depression Treatment Response using Individual Symptoms and Latent Factors with @clairegillan.bsky.social fresh up on @medrxivpreprint.bsky.social www.medrxiv.org/content/10.1...

Predictive Modelling of Depression Treatment Response using Individual Symptoms and Latent Factors

Machine learning models have increasingly been used to identify predictors of treatment response in depression, and it is hoped that they may eventually help with clinical decision making. However, th...

www.medrxiv.org

2 4 9

Claire Gillan @clairegillan.bsky.social · Jul 28

@davluque.bsky.social,
@drrodebaugh.bsky.social

1 3

Claire Gillan @clairegillan.bsky.social · Jul 28

New preprint from the lab on habit learning in compulsivity from the fantastic @kellydonegan.bsky.social! ✅ N=1000, ✅ pre-registered, ✅ within-person crossover design...we study individual differences in habit learning using the Neureka App

Kelly Donegan @kellydonegan.bsky.social · Jul 28

🚨PREPRINT ALERT🚨

Why do some people get stuck doing things that no longer make sense, or are even harmful? Our new study shows compulsivity isn’t just about weakened goal-directed control; people high in compulsivity also form stronger habits

📄: osf.io/preprints/ps...

#PaperFairy #Habits #CogSci

OSF

osf.io

1 5 21

Reposted by Claire Gillan

Kelly Donegan @kellydonegan.bsky.social · Jul 28

🚨PREPRINT ALERT🚨

Why do some people get stuck doing things that no longer make sense, or are even harmful? Our new study shows compulsivity isn’t just about weakened goal-directed control; people high in compulsivity also form stronger habits

📄: osf.io/preprints/ps...

#PaperFairy #Habits #CogSci

OSF

osf.io

2 16 57

Claire Gillan @clairegillan.bsky.social · Jul 28

have not read it - will take a look!

1 1

Reposted by Claire Gillan

Amy Orben @orbenamy.bsky.social · Jul 16

I'm hiring 2+ 2.5-year postdocs over the next few months.

Job 1: looking for someone with deep expertise in computational modelling (reinforcement learning, agent based modelling) on real-world/complex data (closing: 18 August, starting: October-December), www.jobs.cam.ac.uk/job/52059/

2 38 35

Claire Gillan @clairegillan.bsky.social · Jul 15

@davluque.bsky.social @drrodebaugh.bsky.social if you zoom in ;)

1 2

Reposted by Claire Gillan

Vanessa Teckentrup @glassybrain.bsky.social · Jul 15

You happen to be at #CPConf2025 and interested in habitual processes and how they contribute to compulsivity? We've gathered data from 180 individuals across three different compulsion types including phone-checking (not reflected in the DSM!). Come visit me at poster 2.46!

1 3 21

Reposted by Claire Gillan

Kelly Donegan @kellydonegan.bsky.social · Jul 15

Come to my fan-tastic poster at #CPConf2025 🪭 Poster 35 to find out all things habits, gamification or to get a quick cool down 🥵

2 1 26

Claire Gillan @clairegillan.bsky.social · Jul 15

I'd say we're talking 2 weeks to preprint... and all will be revealed

2

Reposted by Claire Gillan

Annika Boldt @annikaboldt.bsky.social · Jul 14

Our new study published in eLife (collaboration with @clairegillan.bsky.social, @celinef.bsky.social and @samgilbert.bsky.social) explores the complex relationship between metacognition, compulsivity and the utilization of external reminders. (1/8)

AI generated image of a wall of post-it notes.

7 4 11

Claire Gillan @clairegillan.bsky.social · Jul 14

Finally, Paveen Phon-Amnuaisuk will present unpublished data examining within-person fluctuations of metacognition over 8 weeks via... you guessed it... the neureka app again 📱

3

Claire Gillan @clairegillan.bsky.social · Jul 14

@glassybrain.bsky.social (Vanessa Teckentrup) will present "Understanding habitual properties of compulsivity using moment-to-moment experience sampling in daily life" with 6 weeks of EMA data tracking OCD, Binge-Eating and Phone-checking compulsions!📱https://osf.io/preprints/psyarxiv/yvhua_v1

1 1 5

Claire Gillan @clairegillan.bsky.social · Jul 14

@kellydonegan.bsky.social will present "Compulsivity is associated with an increase in stimulus- response habit learning"📱,which we are 🤏 close to pre-printing. So 🙏 give us your thoughts and ideas!

3 1 8

Claire Gillan @clairegillan.bsky.social · Jul 14

Sad to be missing #CPConf2025 but there is spirit and via posters from @kellydonegan.bsky.social (TUES) @glassybrain.bsky.social (TUES) and Paveen Phon-Amnuaisuk (WED)

1 1 14

Claire Gillan @clairegillan.bsky.social · Jul 14

Serious FOMO here #CPConf2025

Tobias Hauser @tobiasuhauser.bsky.social · Jul 14

Here we go - #CPConf2025 is on! Fabulous to see so many people attending,n 🥳🥨

1 1 5

Reposted by Claire Gillan

Parnian Rafei @parnianrafei.bsky.social · Jul 8

🚨 New paper alert!
Do time constraints reveal habitual behaviour? 🤔 We directly compared two major paradigms in the habit research field (Outcome Devaluation vs Response Remapping) under identical training and forced-response conditions 🧠 ⏲️ ⌨️

preprint: osf.io/preprints/ps...

OSF

osf.io

1 9 15

Reposted by Claire Gillan

Parnian Rafei @parnianrafei.bsky.social · Jul 2

Hello #Bluesky!👋 I’m late to the party, but excited to join! 🥳Sharing a recent moment at #RLDM2025: presented a poster on my recent PhD project on measuring time-dependent habits in the lab 🧠 – shared fresh findings, and discovered brilliant research while helping the organisers behind the scenes! 👩‍💻

1 3 21

Reposted by Claire Gillan

Matan Mazor @matanmazor.bsky.social · Jul 1

Claire and team did a remarkable job here, and you should read their reply. An updated preprint from us will be out shortly, reflecting our revised understanding of these issues.

Claire Gillan @clairegillan.bsky.social · Jul 1

We @smfleming.bsky.social, Marion Rouault and @seowxft.bsky.social and I) have posted a reply osf.io/preprints/ps... to a preprint that recently raised concerns about the validity of associations between mental health and metacognition from online studies. I hope you can take the time to read it.

OSF

osf.io

1 5 11

Claire Gillan @clairegillan.bsky.social · Jul 1

and by published - i mean pre-printed!! 🙃

1

Claire Gillan @clairegillan.bsky.social · Jul 1

We have now published a formal reply to this preprint. Please do check it out to hear our perspective on these findings and see new analyses that we believe speak to the robustness of the phenomenon osf.io/preprints/ps...

OSF

osf.io

1 1

Claire Gillan @clairegillan.bsky.social · Jul 1

We want to thank @noamsarna.bsky.social @matanmazor.bsky.social and @ruvidar.bsky.social for their feedback on this reply and their productive discussions throughout. I think they've raised an important issue, but I hope we don't throw the baby out with the bathwater.

1 1

Claire Gillan @clairegillan.bsky.social · Jul 1

So what about those patient studies showing lower confidence in OCD that started this whole conversation? We think the issue is not between online versus in-lab studies, but between having a diagnosis of OCD and scoring high on a dimension of CIT. We unpack this in the paper.

1 1

Claire Gillan @clairegillan.bsky.social · Jul 1

In our reply, we show that the FA solution is replicated in symptomatic patients with a different pattern of skew to m-turkers. We also show a near identical FA solution for two samples that differ in attentiveness... mturk vs. unpaid citizen scientists from the Neureka app (below).

1