Lightnews — Scholar-powered news

Giacomo Bignardi @bignardi.bsky.social · 3h

I've added a new example to our paper's repo, demonstrating how our reliability method replicates Cronbach's alpha for a simple model, but also how our method can account for: (i) binary data, (ii) varying numbers of items per pps, & (iii) improvement over trials www.bignardi.co.uk/8_bayes_reli...

2

Giacomo Bignardi @bignardi.bsky.social · 7d

Thanks Edwin! I only learned from the best 😜

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

Great! Sorry just saw this before my replies :)

Giacomo Bignardi @bignardi.bsky.social · 7d

I think both methods should converge to the same answer as the number of draws -> ∞. I think your approach is 1-V_w/V whereas empirical reliability is defined as v_a/(v_a+v_w) - but it should be the same as draws -> ∞. Pic from onlinelibrary.wiley.com/doi/10.1002/...

Giacomo Bignardi @bignardi.bsky.social · 7d

Hi Ruben - i had the same thought - and it wasn't easy finding the reference in an old IRT manual! In your post, do you divide the variance of subjects' posterior means by the total variance in MCMC draws across subjects? The best description of ER is in here by Phil Chalmers tinyurl.com/4f3vv5e8

Difference between empirical and marginal reliability of an IRT model

I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the

tinyurl.com

1

Giacomo Bignardi @bignardi.bsky.social · 7d

...

Giacomo Bignardi @bignardi.bsky.social · 7d

Any (ideally helpful) comments/feedback/critique welcome! Email at: [email protected]

1

Giacomo Bignardi @bignardi.bsky.social · 7d

Alongside the preprint osf.io/h54k8 I've uploaded an example of how to calculate RMU with go/no-go task data, comparing RMU with split-half and test-retest reliability estimates www.bignardi.co.uk/8_bayes_reli... - more software support to come if there's demand.

Tutorial: Calculating RMU reliability for a go/no-go task

www.bignardi.co.uk

1 3

Giacomo Bignardi @bignardi.bsky.social · 7d

In the paper, we also demonstrate that our method yields a similar point estimate to using the ratio of subjects' posterior mean variance (PMV) divided by PMV + average posterior variance. However, our approach also provides credible intervals, which are essential when sample sizes are small.

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

We also compared RMU to coefficient alpha and H (Study 1) and split-half reliability (Study 2). RMU generally had lower bias and error, as well as better coverage. However, the main benefit of RMU is that it can be applied across a wide range of Bayesian measurement models - unlike alpha.

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

We ran 3 simulation studies to evaluate RMU's bias, accuracy and coverage (% of times the 95% credible intervals included the right answer) across linear factor, signal detection theory and reinforcement learning models. Results were good-to-great across studies and simulation conditions.

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

Our method involves taking two random draws (without replacement) from each subject's posterior and computes the correlation between draws. We repeat this process many times and use the distribution of correlations to calculate credible intervals or posterior probabilities (e.g., reliability > .70).

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

The intuition behind our method is that the amount of overlap between posteriors is important: in the example above, if the posteriors became significantly broader, we would be much more uncertain in each estimate and struggle to distinguish between participants, reducing reliability.

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

Our reliability method can be applied to most fitted Bayesian measurement models. In the example below, I've used a Bayesian signal detection model to estimate posterior distributions for each subject's performance on a go/no-go test. Each curve shows our certainty in each subject's estimate.

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

While estimating reliability with questionnaires is straightforward, there are few methods when deriving individual differences from computational models. Thus, reliability is often poor for widely used computational measures when evaluated with retest data (e.g., pubmed.ncbi.nlm.nih.gov/36940888/ )

Individual differences in computational psychiatry: A review of current challenges - PubMed

Bringing precision to the understanding and treatment of mental disorders requires instruments for studying clinically relevant individual differences. One promising approach is the development of com...

pubmed.ncbi.nlm.nih.gov

1 1

Giacomo Bignardi @bignardi.bsky.social · 7d

New preprint with @rogierk.bsky.social @paulbuerkner.com - we introduce "relative measurement uncertainty" - a reliability estimation method that's applicable across a broad class of Bayesian measurement models (e.g., generative-, computational- and item response theory-models osf.io/h54k8

OSF

osf.io

1 4 11

Giacomo Bignardi @bignardi.bsky.social · 14d

...

Giacomo Bignardi @bignardi.bsky.social · 14d

Inspecting the regression coefficients, we found that going to bed hungry was the strongest and most consistent predictor out of all the food insecurity questions.❗We should note that the survey took place during school term, and food insecurity is likely even worse during the school holidays.

1

Giacomo Bignardi @bignardi.bsky.social · 14d

📈Comparing model predictions for low food insecurity to medium or high food insecurity, we found moderate (.3-.5 standard deviations) changes in depression and adolescent wellbeing scores, and smaller changes (.2-.3 standard deviations) in anxiety, loneliness and positive thoughts.

1 1

Giacomo Bignardi @bignardi.bsky.social · 14d

Using #marginaleffects, we compared model predictions for each outcome variable across three levels of food insecurity: "low", "medium", and "high". Each level represents an "average" participant who either responded "never", "sometimes", or "often" to each food insecurity question.

1

Giacomo Bignardi @bignardi.bsky.social · 14d

Next, we used ordinal regression models to estimate the association between the three food insecurity questions and six mental health outcomes, controlling for demographics. We modelled the effect of each food insecurity question separately (and modelled "sometimes" and "often" separately).

1

Giacomo Bignardi @bignardi.bsky.social · 14d

We found 4-5% of CYP sometimes experience food insecurity, and 1-2% often experience food insecurity, across our three questions (F1: food bank usage, F2: food affordability, F3: going to bed hungry). Post-stratification weighting using school census data did not massively alter these estimates.

Table showing the prevalence of food insecurity responses across three questions, split into "never", "sometimes" and "often" responses.

1

Giacomo Bignardi @bignardi.bsky.social · 14d

📣New study 📣 We examined the prevalence of food insecurity and its association with mental health and wellbeing, using the 2023 @oxwell.bsky.social survey of 38,430 children and young people aged 8–19 in England with @sjblakemore.bsky.social @minafazeloxford.bsky.social doi.org/10.1002/jcv2...

JCPP Advances | ACAMH Child Development Journal | Wiley Online Library

BackgroundWe aimed to assess the prevalence of food insecurity reported by children and young people in four areas of England in 2023 and examine its association with mental health and wellbeing. Met...

doi.org

1 3 11

Reposted by Giacomo Bignardi

Prof Sarah-Jayne Blakemore @sjblakemore.bsky.social · Aug 31

New paper! Excellent work by @bignardi.bsky.social and @minafazeloxford.bsky.social, who leads the Oxwell study

The prevalence of food insecurity and its relationship with wellbeing in a large, cross-sectional study of children and young people in England

Out soon. Preprint:
osf.io/preprints/ps...

OSF

osf.io

7 19

Reposted by Giacomo Bignardi

Thalia Eley @thaliaeley.bsky.social · Aug 7

Delighted to be hiring a new Clinical Psychology Lecturer to come and join our vibrant environment at the SGDP Centre @kingsioppn.bsky.social @kingscollegelondon.bsky.social please share!

my.corehr.com/pls/coreport...

my.corehr.com

8 6