Lightnews — Scholar-powered news

Noah Haber @whaleactually.com · Sep 2

To the eight of you out there whose brains are also damaged in this particular way, thank you/I am sorry.

1

Noah Haber @whaleactually.com · Sep 2

"Elapsed time" is to Intention to Treat as "Moving time" is to Per-Protocol.

I will not explain any further.

1 1

Noah Haber @whaleactually.com · Aug 6

It's giving Borne vibes....

2

Noah Haber @whaleactually.com · Jul 31

A few of the first recruitment attempts to US-based journals have been met with a "times are too uncertain to take on a project like this," as expected

Small potatoes in the scheme of things, but just another example of the astounding destruction of scientific progress happening right now.

Noah Haber @whaleactually.com · Jul 28

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.

This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.

Now piloted and ready for the main stage, and looking for partners! 🧵👇

4

Noah Haber @whaleactually.com · Jul 29

I am become randomista, bringer of exogeneity.

2

Noah Haber @whaleactually.com · Jul 29

The "meta trial" idea is maybe the most ambitious thing I've ever gotten to try in real life.

In theory, it goes a LONG way towards solving the feasibility, logistics, and incentives problems inherent in multi-unit policy experiments at scale.

Not just for journal policy. For policy period.

Noah Haber @whaleactually.com · Jul 28

What we would really want in the end is a BUNCH of compatible, but realistic and pragmatic trials of custom variations on this policy and get to wrap it all up in a nice meta analysis.

For the last year or so, we've been piloting a pretty wild approach to getting exactly that:

A study-in-a-kit

1 1

Noah Haber @whaleactually.com · Jul 29

Yep, exactly. In theory, it's a win all around. More incentive for partners to collaborate, more flexibility for partners, more realistic policy roll outs, etc.

No reason to limit this to journal experiments either; the same idea can be done for any multi-unit policy experiment.

1 1

Reposted by Noah Haber

Joachim Vandekerckhove 🏖️ @joachim.cidlab.com · Jul 29

This meta trial idea is pretty cool. Eventually you could arrive at a situation where journals help researchers get individual credit for research labor on large collaborative projects

Noah Haber @whaleactually.com · Jul 28

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.

This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.

Now piloted and ready for the main stage, and looking for partners! 🧵👇

1 1 1

Reposted by Noah Haber

Noah Haber @whaleactually.com · Jul 28

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.

This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.

Now piloted and ready for the main stage, and looking for partners! 🧵👇

2 13 21

Noah Haber @whaleactually.com · Jul 28

And of course, typos are my own, and not those of the NSF which funds this project (grant #2152424) nor the IRB who approved it (University of Virginia Institutional Review Board Protocol #6358).

Noah Haber @whaleactually.com · Jul 28

@cos.io also recently launched a journal that puts a whole project from conception to publication (including multi-stage review) in one place called the Lifecycle Journal.

If Registered Revisions is like a mini registered report, Lifecycle Journal is like a mega one.

lifecyclejournal.org

Lifecycle Journal | Adding trust to your research, from conception through completion

lifecyclejournal.org

5 5

Noah Haber @whaleactually.com · Jul 28

Yep, if I am interpreting you correctly, those are usually called "Registered Reports." There are quite a few journals that have that as an option, but takeup from authors is pretty low.

Registered *revisions* is a more specific take on that idea.

2 1

Noah Haber @whaleactually.com · Jul 28

Now we're ramping up to the main phase of the project and gathering journal partners.

Are you a potentially interested journal editor? Know someone who might be? Let's chat!

Feel free to DM me here or email me ([email protected]).

More info: www.cos.io/r3ct/registe...

Impact of Registered Revisions

Publication pre-commitment devices such as Preregistration, Registered Reports, and Registered Revisions may substantially reduce publication biases, prepublication biases (e.g. p-hacking and HARKING)...

www.cos.io

1

Noah Haber @whaleactually.com · Jul 28

It's sorta like someone smashed together a multi-center trial with a prospective meta analysis.

If this works, it's a potentially game-changing way to do large scale policy evidence generation.

And we have a good idea that it DOES work, because we've been piloting it with six journal partners.

1

Noah Haber @whaleactually.com · Jul 28

That means you also get to publish your own results. That might be nice for the journal, but could be extra nice for, say, a junior editor who leads it.

The "trick" here is that we use those data in a pre-planned meta analysis in a couple years (with coathorship, of course).

1

Noah Haber @whaleactually.com · Jul 28

The "kit" is a package for journals that includes things like:

Protocols with flexibility for variation
A data collection infrastructure
An IRB pathway (pre approved for most)
Data cleaning
Suggested code
A support community

But the best part is this: if you run the trial, you own the trial.

1

Noah Haber @whaleactually.com · Jul 28

What we would really want in the end is a BUNCH of compatible, but realistic and pragmatic trials of custom variations on this policy and get to wrap it all up in a nice meta analysis.

For the last year or so, we've been piloting a pretty wild approach to getting exactly that:

A study-in-a-kit

1 1

Noah Haber @whaleactually.com · Jul 28

Even bigger question:

How on earth are you going to get an experiment large enough with a bunch of diverse journals, coordinated together with the exact same protocols, on the same exact timelines, and producing actionable evidence?

You can't.

Which is where the whole "Meta Trial" thing comes in.

1

Noah Haber @whaleactually.com · Jul 28

Then the authors go execute their plan. As long as they followed the plan and the other revisions are sufficiently addressed, the paper is accepted, regardless of the new results.

But how does that impact timelines? Questionable research practices? Author experience?

¯\_(ツ)_/¯

1

Noah Haber @whaleactually.com · Jul 28

When an author gets a "do something new" comment, editors as the authors to address it by giving a summary of what they plan to do to address the comment.

The editors (and maybe reviewers) then decide whether that plan is acceptable to address the issue, and issue an in principle acceptance.

1

Noah Haber @whaleactually.com · Jul 28

It leaves a lot of uncertainty for what authors should do, what peer reviewers expect, pressure for questionable research practices to slip in to prevent late rejection for "wrong" results, etc.

Bad times.

Registered Revisions aims to address that through Revision Plan and in principle acceptance.

1 1

Noah Haber @whaleactually.com · Jul 28

What's a "Registered Revision" policy?

The idea is similar to registered reports, but it occurs during standard peer review.

You've all seen peer review comments like "hey this stat is bad can you run something else?" or "can you collect more samples to text X?"

Those are tough to deal with.

1

Noah Haber @whaleactually.com · Jul 28

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.

This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.

Now piloted and ready for the main stage, and looking for partners! 🧵👇

2 13 21

Noah Haber @whaleactually.com · Jul 10

only the sith deal in absolute attribution

4

Noah Haber @whaleactually.com · Jul 10

death to "is it A or is it B that causes Y"

long live "how much do A and B contribute to Y"

1 6