Lightnews — Scholar-powered news

Reposted by Clintin Davis-Stober

Richard D. Morey @richarddmorey.bsky.social · 13d

Simonsohn has now posted a blog response to our recent paper about the poor statistical properties of the P curve. @clintin.bsky.social and I are finishing up a less-technical paper that will serve as a response. But I wanted to address a meta-issue *around* this that may clarify some things. 1/x

Data Colada @datacolada.bsky.social · 14d

Would p-curve work if you dropped a piano on it?
datacolada.org/129

PIano being dropped on car in car testing facility

2 29 72

Reposted by Clintin Davis-Stober

Berna Devezer @devezer.bsky.social · Aug 14

I love this post about science and metascience. A lot of quotables but I’ll lead with this:

“Those seeking a scientific method – one that can be written down and followed mechanically […] – betray a kind of childish impatience with a process they clearly don’t understand.”

John Kennedy @micefearboggis.bsky.social · Aug 13

This is sort of part two to the far more enjoyable (to write and, hopefully, to read) essay "how to science a science with science". You might want to read that first.

diagrammonkey.wordpress.com/2025/07/03/h...

How to science a science with science

The purest description of the scientific method I ever saw was in a novel by Kurt Vonnegut. A workman discovers that if he puts a bucket full of nuts and bolts on one of the many supporting struts …

diagrammonkey.wordpress.com

2 7 36

Reposted by Clintin Davis-Stober

Iris van Rooij 💭 @irisvanrooij.bsky.social · Aug 12

AI slop and the destruction of knowledge irisvanrooijcogsci.com/2025/08/12/a...

AI slop and the destruction of knowledge

This week I was looking for info on what cognitive scientists mean when they speak of ‘domain-general’ cognition. I was curious, because the nuances are relevant for something I am researching at t…

irisvanrooijcogsci.com

20 240 490

Reposted by Clintin Davis-Stober

Carl T. Bergstrom @carlbergstrom.com · Aug 12

1. "'Trusting the experts is not a feature of either a science or democracy," Kennedy said."

It's literally a vital feature of both science and of representative democracy.

I've written a fair bit about trust in expertise as a vital mechanism in the collective epistemology of science.

RFK Jr. in interview with Scripps News: ‘Trusting the experts is not science’

HHS Secretary RFK Jr. sat down with Scripps News for a wide-ranging interview, discussing mRNA vaccine funding policy changes and a recent shooting at the Centers for Disease Control and Prevention.

www.scrippsnews.com

540 2.8K 10K

Reposted by Clintin Davis-Stober

Iris van Rooij 💭 @irisvanrooij.bsky.social · Aug 11

We proved they could not scale link.springer.com/article/10.1...

4 24 75

Clintin Davis-Stober @clintin.bsky.social · Aug 10

Definitely something worth digging into. I’ll give it some thought

1

Reposted by Clintin Davis-Stober

Berna Devezer @devezer.bsky.social · Aug 10

here was our call for methodological standards in metaresearch four years ago. instead of getting fixated on a particular inference we'd like to make, we need to maintain scientific standards, do the hard work, respect the evidence. we can't keep jumping at self-serving solutions without question.

The case for formal methodology in scientific reform | Royal Society Open Science

Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological ref...

share.google

1 4 21

Reposted by Clintin Davis-Stober

Berna Devezer @devezer.bsky.social · Aug 10

some of the discussion around the p-curve paper is depressing. i still see many missing the point clearly stated in the conclusion, and instead of demanding strong standards or questioning whether they're even asking good questions, they've immediately started asking for replacement methods.

3 7 30

Clintin Davis-Stober @clintin.bsky.social · Aug 10

I would add that while pcurve is comprised of tests, that indeed correspond to error rates, the actual hypotheses being tested have little to do with “evidential value”

1

Clintin Davis-Stober @clintin.bsky.social · Aug 10

Test statistics being used are just simple sums, no 3rd moment information enters the test.

1 2

Clintin Davis-Stober @clintin.bsky.social · Aug 10

Would this be grounds for dismissing the remaining 54 studies as lacking value? It makes no sense. Part of the problem is that the original p curve papers aren’t clear on what exactly is being tested. The authors claim they are tests of skew, but this is incorrect as the

1 2

Clintin Davis-Stober @clintin.bsky.social · Aug 10

Happy to clarify. Pcurve is used to test whether a set of studies have (or lack) “evidential value” (which is not really defined). But the actual hypotheses being tested by pcurve don’t permit this, as Richard and I show. Suppose one study WAS underpowered in a set of 55 studies -

1 1

Clintin Davis-Stober @clintin.bsky.social · Aug 9

All pcurve tests are just a simple sum of transformed pvalues. There is a fundamental disconnect between the null hypotheses being tested by p-curve and the claims being made.

1 2 14

Clintin Davis-Stober @clintin.bsky.social · Aug 9

What this means is that a significant result for either test only allows one to claim that “at least one” study (out of the set) doesn’t have the property being considered. Why does this happen? Because pcurve completely ignores the configuration of the pvalues being considered.

1 5

Clintin Davis-Stober @clintin.bsky.social · Aug 9

The test for evidential value simply examines whether the effect size is zero for all studies. The test for lack of evidential value tests whether all studies are “underpowered”, I.e., have small non-centrality parameters.

1 5

Clintin Davis-Stober @clintin.bsky.social · Aug 9

The developers of p-curve claim that p-curve can be used to make claims about the evidential value (or lack thereof) of whole sets of studies. We show that the actual hypotheses being tested do not allow for such strong conclusions.

1 4

Clintin Davis-Stober @clintin.bsky.social · Aug 9

The basic idea of p-curve rests on the idea that the skew of a set of p-values is informative about whether QRPs are occurring. As we show, the p-curve tests have nothing to do with skew. It is trivial to create left skewed pvalues that p-curve would confidently label as right skewed.

1 2 11

Clintin Davis-Stober @clintin.bsky.social · Aug 9

New paper with @richarddmorey.bsky.social now out in JASA, where we critically examine p-curve. Below is Richard’s excellent summary of the many poor statistical properties of p-curve (with link to paper). I wanted to add some conceptual issues that we also tackle in the paper.

Richard D. Morey @richarddmorey.bsky.social · Aug 8

Paper drop, for anyone interested in #metascience, #statistics, or #metaanalysis! @clintin.bsky.social and I show in a new paper in JASA that the P-curve, a popular forensic meta-analysis method, has deeply undesirable statistical properties. www.tandfonline.com/doi/full/10.... 1/?

Cover page for the manuscript: Morey, R. D., & Davis-Stober, C. P. (2025). On the poor statistical properties of the P-curve meta-analytic procedure. Journal of the American Statistical Association, 1–19. https://doi.org/10.1080/01621459.2025.2544397

Abstract for the paper: The P-curve (Simonsohn, Nelson, & Simmons, 2014; Simonsohn, Simmons, & Nelson, 2015) is a widely-used suite of meta-analytic tests advertised for detecting problems in sets of studies. They are based on nonparametric combinations of p values (e.g., Marden, 1985) across significant (p < .05) studies and are variously claimed to detect “evidential value”, “lack of evidential value”, and “left skew” in p values. We show that these tests do not have the properties ascribed to them. Moreover, they fail basic desiderata for tests, including admissibility and monotonicity. In light of these serious problems, we recommend against the use of the P-curve tests.

2 20 52

Reposted by Clintin Davis-Stober

Daniel Heck @danielheck.bsky.social · Jun 12

New paper by my PhD student @semihaktepe.bsky.social now published 🚀

"Revisiting the effect of discrepant perceptual fluency on truth judgments" tinyurl.com/2eepue5y

--> Two experiments & a meta-analysis indicate that high visual contrast does not lead to higher truth judgments.

3 14

Clintin Davis-Stober @clintin.bsky.social · Jun 11

I'm so sorry this happened to you. There is no excuse for such bs.

1 3

Reposted by Clintin Davis-Stober

Berna Devezer @devezer.bsky.social · May 31

wherein I'm quoted with "Science is essentially effective to the extent that it can remain independent, decentralized, and democratic." (gift link, email required)

Why Trump’s push for ‘gold-standard science’ has researchers alarmed

Many scientists fear the Trump administration’s new standard means putting political appointees in charge, which could undercut independent research.

wapo.st

3 22 87

Reposted by Clintin Davis-Stober

Daniel Heck @danielheck.bsky.social · May 28

🚀Postdoc position @unimarburg.bsky.social in the project:

"Bridging the Gap Between Verbal Psychological Theories & Formal Statistical Modeling with Large Language Models"
(funded by @volkswagenstiftung.de)

📅Start: 01.10.2025 | ⏳4 years
🔗 Apply now: uni-marburg.de/jhbCen
🔄 Thanks for sharing!

Postdoc

uni-marburg.de

1 23 28

Reposted by Clintin Davis-Stober

Iris van Rooij 💭 @irisvanrooij.bsky.social · May 14

NEW paper! 💭🖥️

“Combining Psychology with Artificial Intelligence: What could possibly go wrong?”

— Brief review paper by @olivia.science & myself, highlighting traps to avoid when combining Psych with AI, and why this is so important. Check out our proposed way forward! 🌟💡

osf.io/preprints/ps...

Table 1
Typology of traps, how they can be avoided, and what goes wrong if not avoided. Note that all traps in a sense constitute category errors (Ryle & Tanney, 2009) and the success-to-truth inference (Guest & Martin, 2023) is an important driver in most, if not all, of the traps.

11 69 220

Clintin Davis-Stober @clintin.bsky.social · May 20

Great pic

1

Reposted by Clintin Davis-Stober

Richard D. Morey @richarddmorey.bsky.social · May 15

If you'll be at APS2025 in DC next week, I'll be talking about the terrible statistical properties of the p curve procedure in the "Current Issues in Meta-Science" session Saturday, May 24th at 3pm. This will likely be my last US conference in a very long time. A brief summary follows. 1/

Screenshot of the APS program for "Current Issues in Meta-Science", Saturday, May 24, 3:00pm

Titles and authors in the session:

* Statistical Power in the Light of Methodological Reform (Jolynn Pek)

* The Poor Statistical Properties of the P-Curve Procedures (Richard Morey)

* Consistent Methods Protect Against False Findings Produced By p Hacking (Duane Wegener)

* Accumulating Evidence across Studies (Blakeley McShane)

2 18 47