Lightnews — Scholar-powered news

Jamie Cummins

@jamiecummins.bsky.social

2.7K followers 660 following 870 posts

Currently a visiting researcher at Uni of Oxford. Normally at Uni of Bern. Meta-scientist building tools to help other scientists. NLP, simulation, & LLMs. Creator and developer of RegCheck (https://regcheck.app). 1/4 of @error.reviews. 🇮🇪

regcheck.app).

Posts Media Videos Starter Packs

Pinned

Jamie Cummins @jamiecummins.bsky.social · Jul 23

Introducing RegCheck: a tool which uses Large Language Models to automatically compare preregistered protocols with their corresponding published papers and highlights deviations.

@malte.the100.ci @ianhussey.bsky.social @ruben.the100.ci @bjoernhommel.bsky.social

regcheck.app

RegCheck.app

RegCheck is an AI tool to compare preregistrations with papers instantly.

regcheck.app

5 51 100

Reposted by Jamie Cummins

Jack Wilkinson @jdwilko.bsky.social · 5d

Introductory online INSPECT-SR workshop. November 6th, 12-2pm UK-time. Free, places limited. BOOK: www.trybooking.com/uk/events/la...

Introduction to INSPECT-SR Training Workshop November

An introductory 2-hour online workshop will introduce participants to the INSPECT-SR tool for assessing trustworthiness of randomised controlled...

www.trybooking.com

1 6 5

Jamie Cummins @jamiecummins.bsky.social · 1d

thanks for the mention! 😊

Reposted by Jamie Cummins

Crystal Lewis @cghlewis.bsky.social · 1d

Issue 16 of RDM Weekly is out! 📬

It includes:
- Data is Not Available Upon Request @ianhussey.mmmdata.io
- AI Generated Participants in Social Science @jamiecummins.bsky.social @science.org
- Why’s it Hard to Teach Data Cleaning? @randyau.com
and more!

rdmweekly.substack.com/p/rdm-weekly...

RDM Weekly - Issue 016

A weekly roundup of Research Data Management resources.

rdmweekly.substack.com

2 7 18

Reposted by Jamie Cummins

Beth Popp Berman @epopppp.bsky.social · 4d

Interesting article/paper.

I'm much less anti-AI than a lot of people on my feed. But pretty skeptical it can simulate human behavior effectively for social scientific purposes -- at least in cases where variation among humans, rather than acting like an average human, is what's important.

AI-generated ‘participants’ can lead social science experiments astray, study finds

Data produced by “silicon samples” depends on researchers’ exact choice of models, prompts, and settings

www.science.org

1 6 37

Jamie Cummins @jamiecummins.bsky.social · 3d

WCL winners, they’ll never sing that 😉

Reposted by Jamie Cummins

Ian Hussey @ianhussey.mmmdata.io · 4d

My article "Data is not available upon request" was published in Meta-Psychology. Very happy to see this out!
open.lnu.se/index.php/me...

LnuOpen | Meta-Psychology

open.lnu.se

4 36 100

Jamie Cummins @jamiecummins.bsky.social · 5d

I’ve seen @malte.the100.ci recently using one that looked very cool

1 1

Jamie Cummins @jamiecummins.bsky.social · 6d

OMG I can’t wait to listen!

1 3

Reposted by Jamie Cummins

Saloni @scientificdiscovery.dev · 6d

New episode of HARD DRUGS!

AlphaFold, ProteinMPNN & other AI tools are transforming biology and drug design.

But how do they work? What can’t they do? And can we use them to make a vaccine against Strep A for the very first time?

In this episode, Jacob and I talk about hacking proteins with AI.

Hacking proteins with AI

open.spotify.com

3 9 36

Reposted by Jamie Cummins

Benjamin Paaßen @bpaassen.bsky.social · 7d

Our work on simulated participants also makes a small appearance arxiv.org/abs/2508.06950

Large Language Models Do Not Simulate Human Psychology

Large Language Models (LLMs),such as ChatGPT, are increasingly used in research, ranging from simple writing assistance to complex data annotation tasks. Recently, some research has suggested that LLM...

arxiv.org

1 3

Reposted by Jamie Cummins

Benjamin Paaßen @bpaassen.bsky.social · 7d

@cathleenogrady.bsky.social has just published the story "AI-generated ‘participants’ can lead social science experiments astray, study finds" for Science. It is, once more, a reason to be careful when relying on LLM-generated data in empirical research. www.science.org/content/arti...

AI-generated ‘participants’ can lead social science experiments astray, study finds

Data produced by “silicon samples” depends on researchers’ exact choice of models, prompts, and settings

www.science.org

1 1 4

Jamie Cummins @jamiecummins.bsky.social · 7d

Looking forward to reading this, and I’m glad you’ve written it!

Reposted by Jamie Cummins

Chris Chapman @cchapman.bsky.social · 7d

Excellent 🧵 about LLM synthetic data (silicon samples etc) and why they don't solve any particular problem in human research.

FWIW, in addition to results and considerations like these, I've argued elsewhere that the entire question is ill-formed: quantuxblog.com/synthetic-su...

2 2 5

Jamie Cummins @jamiecummins.bsky.social · 7d

There isn't really a fixed term tbh, people use a few different ones depending on field/domain/preference. Silicon samples seems to be the most common but there are a bunch of others, like synthetic samples/synthetic participants/etc.

Jamie Cummins @jamiecummins.bsky.social · 7d

Couldn't agree more.

Jamie Cummins @jamiecummins.bsky.social · 7d

Clearly I missed my true career-calling as a diplomat lol

Jamie Cummins @jamiecummins.bsky.social · 7d

OMG. Did not catch this one during my lit review. Wow.

Jamie Cummins @jamiecummins.bsky.social · 7d

Starting to feel eerily like Severance....

1 1

Jamie Cummins @jamiecummins.bsky.social · 7d

that should have been my full abstract!

Reposted by Jamie Cummins

Lora Kolodny @lorak.bsky.social · 7d

👀 studying real humans better for understanding humans than not

Jamie Cummins @jamiecummins.bsky.social · 7d

@science.org just dropped a story covering this preprint! Check it out below, and thanks to @cathleenogrady.bsky.social for the great write-up! www.science.org/content/arti...

2 4 22

Jamie Cummins @jamiecummins.bsky.social · 7d

@science.org just dropped a story covering this preprint! Check it out below, and thanks to @cathleenogrady.bsky.social for the great write-up! www.science.org/content/arti...

1 16 36

Jamie Cummins @jamiecummins.bsky.social · 7d

Yeah this paper was hugely inspirational for me!

Reposted by Jamie Cummins

Darren Dahly @statsepi.bsky.social · 7d

Science is grounded in observation. Measurement is a tool for observation. Measurements should be evaluated for validity and reliability/uncertainty. Scientists who use measurements without understanding their properties are not really scientists at all.

Jamie Cummins @jamiecummins.bsky.social · 20d

Can large language models stand in for human participants?
Many social scientists seem to think so, and are already using "silicon samples" in research.

One problem: depending on the analytic decisions made, you can basically get these samples to show any effect you want.

THREAD 🧵

The threat of analytic flexibility in using large language models to simulate human data: A call to attention

Social scientists are now using large language models to create "silicon samples" - synthetic datasets intended to stand in for human respondents, aimed at revolutionising human subjects research. How...

arxiv.org

6 16 55

Reposted by Jamie Cummins

Julia M. Rohrer @dingdingpeng.the100.ci · 7d

A lot of psych is already conducted with online convenience samples & ppl are probably excited about silicon samples bc it would allow them to crank out more studies for even less 💸

How about we reconsider the idea that sciencey science involves collecting own data.
www.science.org/content/arti...

AI-generated ‘participants’ can lead social science experiments astray, study finds

Data produced by “silicon samples” depends on researchers’ exact choice of models, prompts, and settings

www.science.org

13 71 230

Jamie Cummins @jamiecummins.bsky.social · 8d

Forget running DOOM on your calculator; someone created a 5 million parameter language model in Minecraft. www.youtube.com/watch?v=VaeI...

I built ChatGPT with Minecraft redstone!

YouTube video by sammyuri

www.youtube.com

2 5