Much thanks to the amazing team: @martonaronvarga.bsky.social , @donvanraven.bsky.social , @kekecszoltan.bsky.social, @jimgrange.bsky.social, @balazsaczel.bsky.social & Máté Gyurkovics

We think that research fields where notable "ground truth" effects are investigated (such as the CSE), a similar systematic exploration of the analytical space is necessary to inform the field's community about common arbitrary decision combinations that can lead to higher false findings.

Based on these results we think that the risks of multiple testing (even with common corrections) are higher than expected, thus sticking to a preregistered analytical protocol is immensely recommended.

in repeated-measures ANOVAs, FPRs were not affected by outlier filtering methods; thus, when severe outlier filtering is justified, repeated-measures ANOVA is a recommended choice for hypothesis testing.

In linear models, type I error rates also increase proportionally to the severity of outlier filters. This inflation of FPR poses a significant risk of false findings; therefore, we do not recommend to use linear mixed models along with severe outlier exclusion techniques, especially on skewed data.

Results showed that certain analytical choice combinations (outlier filtering; data transformation; hypothesis testing method) led to highly inflated false positive rates (type I error rates). Decision pathways where linear mixed-effect models were used were especially impacted.

I am happy to announce the publication of our new work on the impact of arbitrary analytical choices on type I and type II error rates. We simulated reaction time data in a conflict task and analyzed the notable CSE effect in a multiverse manner. Worrying results:
link.springer.com/article/10.3...