Lightnews — Scholar-powered news

Reposted by Noam Sarna

Matan Mazor @matanmazor.bsky.social · Jul 1

Claire and team did a remarkable job here, and you should read their reply. An updated preprint from us will be out shortly, reflecting our revised understanding of these issues.

Claire Gillan @clairegillan.bsky.social · Jul 1

We @smfleming.bsky.social, Marion Rouault and @seowxft.bsky.social and I) have posted a reply osf.io/preprints/ps... to a preprint that recently raised concerns about the validity of associations between mental health and metacognition from online studies. I hope you can take the time to read it.

OSF

osf.io

1 5 11

Noam Sarna @noamsarna.bsky.social · Mar 22

We showed that the AD factor loads highly on reversed items (Figure 6). So, a participant with negative acquiescence will likely score high on the AD factor (as low ratings for reversed items translate to high symptom) and will also tend to endorse low confidence ratings (Figure 1, panel B).

1

Noam Sarna @noamsarna.bsky.social · Mar 22

I think so, yes. As acquiescence is a tendency to give (high/low) ratings irrespective of content. In our paper we show that acquiescence is correlated with confidence ratings but also with scores on the OCI-R for example.

Noam Sarna @noamsarna.bsky.social · Mar 22

Many thanks for sharing this Oliver! It's very interesting and compelling.

1

Noam Sarna @noamsarna.bsky.social · Mar 22

Thanks so much, Claire! Really appreciate your collaborative spirit.

1

Noam Sarna @noamsarna.bsky.social · Mar 17

None of this would have been possible without the open data, code sharing, and immensely useful feedback from @smfleming.bsky.social, @clairegillan.bsky.social, Marion Rouault, and @seowxft.bsky.social, which we are grateful for. [end]

1 5

Noam Sarna @noamsarna.bsky.social · Mar 17

We present some ideas for how future studies may better control for these nuisance effects when testing for correlations between mental health and metacognition. [13/14]

1 3

Noam Sarna @noamsarna.bsky.social · Mar 17

Importantly, our claim is *not* that true correlations between mental health and decision confidence do not exist. But more work is needed to tease apart true effects from nuisance correlations of confidence with surface-level properties of questionnaire-filling behaviour. [12/14]

1 1 3

Noam Sarna @noamsarna.bsky.social · Mar 17

Lastly, we show that the commonly used factor structure of mental health (elifesciences.org/articles/11305) is largely shaped by these two surface-level properties. [11/14]

1

Noam Sarna @noamsarna.bsky.social · Mar 17

We provide direct evidence for these two effects in a new experiment, and use simulations to show that our findings cannot be explained by statistical artefacts. [10/14]

1

Noam Sarna @noamsarna.bsky.social · Mar 17

We then show that inattentive responders tend to be highly confident in their decisions (we have some ideas for why). Together, items that are rarely endorsed (therefore highly skewed) are more positively correlated with decision confidence. [9/14]

1 1

Noam Sarna @noamsarna.bsky.social · Mar 17

And second, we build on Sam Zorowitz’s work (www.nature.com/articles/s41...) and show that, indeed, inattentive responders appear highly symptomatic in items that are rarely endorsed (like item OCI-R 16) [8/14]

1 1 2

Noam Sarna @noamsarna.bsky.social · Mar 17

In line with this surface-level account, we find that, in two published datasets, correlations between poor mental health and confidence are positive for standard items (where a high rating = poor mental health), but negative for reversed items (where a low rating = poor mental health). [7/14]

1 1 1

Noam Sarna @noamsarna.bsky.social · Mar 17

First, a tendency to give high or low ratings to self-report items translates to confidence ratings. Here is a correlation between endorsement of OCI-R item 16 and participants’ general tendency to give high ratings to neutral items such as “there are relatively few breeds of cats”: [6/14]

1 1

Noam Sarna @noamsarna.bsky.social · Mar 17

We think we now understand what is going on. We show that correlations of confidence with mental health in online studies are largely driven by two surface-level properties of questionnaire-filling behaviour: rating biases and inattentive responding. [5/14]

1 4 5

Noam Sarna @noamsarna.bsky.social · Mar 17

As a clinical psychologist in training, this surprised me. In the clinic, obsessive compulsive patients are known to doubt their own cognition. So much so that OCD has been termed “folie du doute” – a madness of doubt. [4/14]

1 3

Noam Sarna @noamsarna.bsky.social · Mar 17

In recent studies, it has been found that endorsement of this item (and other items that measure compulsivity) is associated with high levels of confidence in perceptual decision-making tasks. We replicated this finding. [3/14]

1 1 2

Noam Sarna @noamsarna.bsky.social · Mar 17

Take a moment to read this item: “I feel that there are good and bad numbers”. This is item number 16 from the OCI-R questionnaire: a list of 18 questions that is used to assess obsessive-compulsive tendencies. [2/14]

1 1

Noam Sarna @noamsarna.bsky.social · Mar 17

Hey Bluesky! I’m excited to share my new preprint with @matanmazor.bsky.social and @ruvidar.bsky.social, where we show that surface-level questionnaire filling behaviours drive correlations between mental health and metacognition (confidence ratings). osf.io/preprints/ps... [1/14]

a close up of a man wearing glasses and a hat with abc family on the bottom

ALT: a close up of a man wearing glasses and a hat with abc family on the bottom

media.tenor.com

4 22 67

Noam Sarna @noamsarna.bsky.social · Mar 17

Importantly, our claim is *not* that true correlations between mental health and decision confidence do not exist. But more work is needed to tease apart true effects from nuisance correlations of confidence with surface-level properties of questionnaire-filling behaviour. [12/14]

Noam Sarna @noamsarna.bsky.social · Mar 17

Lastly, we show that the commonly used factor structure of mental health (elifesciences.org/articles/11305) is largely shaped by these two surface-level properties. [11/14]

1

Noam Sarna @noamsarna.bsky.social · Mar 17

We provide direct evidence for these two effects in a new experiment, and use simulations to show that our findings cannot be explained by statistical artefacts. [10/14]

1

Noam Sarna @noamsarna.bsky.social · Mar 17

We then show that inattentive responders tend to be highly confident in their decisions (we have some ideas for why). Together, items that are rarely endorsed (therefore highly skewed) are more positively correlated with decision confidence. [9/14]

1

Noam Sarna @noamsarna.bsky.social · Mar 17

And second, we build on Sam Zorowitz’s work (www.nature.com/articles/s41...) and show that, indeed, inattentive respondes appear highly symptomatic in items that are rarely endorsed (like item OCI-R 16) [8/14]

1

Noam Sarna @noamsarna.bsky.social · Mar 17

In line with this surface-level account, we find that, in two published datasets, correlations between poor mental health and confidence are positive for standard items (where a high rating = poor mental health), but negative for reversed items (where a low rating = poor mental health). [7/14]

1