I don’t think the problem is that the study is observational (there’s almost nothing else to go on anyway), but it’s plausible that you’d find no difference (even if there was a strong true effect of prereg) when conditioning on high-IF journals because they select more strongly on positive results.

In sum: all global prevalence rates about gaming addiction/disorder seem to be a partial combination of both gaming *and* gambling prevalence. The same applies to other studies, which use gaming measures in relevant languages. Chinese seems to be a major exception.

behold, we found great variation in how people think! Many activities that we thought would be “gaming” weren't & vice versa, eg half of the participants interpreted ‘gambling’ to be ‘gaming’. Ergo: surveying ‘gaming’ without defining it creates data mess

..we then reviewed the next 358 studies to see if we could find more information in papers & public materials. Basically the only case where gambling was excluded was inherent language: ~30% of the studies used Chinese, where confusion shouldn’t exist. Field-wide measurement error is hence ~70%

The raw data is now lost to the sands of time, but a multiverse analysis can still be conducted on the differences between the conditions at post by extracting data from the figures.

Results show no differences in means under any set of analytic choices.

To understand the impact of this flexibility in scoring, we ran a multiverse analysis on an open IGT dataset using 205 of the identified scores. The correlations ranged from -0.942 to 0.998 with a median r = 0.022.

But the most striking finding was in the scoring: We found 244 distinct scores used to analyze IGT outcomes. 177 of those scores appeared only once in our sample, showing a massive fragmentation in how IGT data is analyzed.

Same, plus the methods are usually the most important bit?

FYI to anyone whose "task queued" status never changed - there was a small bug preventing them from being processed, and this is now fixed. Your reports should be ready soon!

8/🧵

Finally, I fulfill something long-promised: a full, open-source release of the entire RegCheck codebase.

This comes with instructions on how to run RegCheck locally, both as a local implementation of the GUI, and as a standalone CLI application. Find the code here:

github.com/JamieCummins...

6/🧵

I have also added integration with the ClinicalTrials.gov API.

You can now provide a ClinicalTrials.gov identifier and RegCheck will automatically retrieve the registration.

5/🧵

Crucially: quote extraction is NOT done by a generative LLM.

Instead, it uses embeddings-based comparisons between text chunks and user-defined comparison dimensions.

This means extraction is deterministic: run RegCheck with the same inputs, you'll get the same quotes as outputs.

Ultimately, we may care more about the specific mechanisms than about isolating period effects. After all, those are what we can intervene on! But we can't really make progress on them just looking at variables over time and suggesting that changes are attributable to one thing or another.