oops, alpha = cronbach's alpha, which is a very common measure of reliability in the psychonetric lit en.m.wikipedia.org/wiki/Cronbac...

because alpha is math equivalent to the average (length adjusted) split half reliability, it fits this use case well. e.g. if you score 4 of 8 items (call this test A score) and then also the left out 4 (test B score), the correlation of test A and B (across test takers) is what alpha measures

yeah sounds like OP is after something similar to the concept of reliability (in the psychometric sense, although not sure what the construct is here)

the whole simulation thing doesn't seem necessary if that's the case—the standard reliability measures will do

Congrats dude!!

Ah yeah most of the work im familiar with skews toward generative models. e.g. like this classic piece, which captures various interactions between Stroop conditions, learning effects, etc.

psycnet.apa.org/record/1990-...

Probably not what you are referring to, but the Stroop task is our main example here: psycnet.apa.org/fulltext/202...

yeah this is a great set

Wish I could have made it in person!

i have now vented on reddit and bsky so maybe i will feel better about it now

tariffs amirite

Thanks for the endorsement! Awesome to hear it's been influential to your work 😁

And yes the KL finding is super interesting by itself, happy to hear someone found it buried in the supplement ha

Thank you! Glad you have found it useful 🤓

it was horrible, there were 10+ papers based on the initial preprint/blog that were in print before this one eventually made it there 😅

Yeah I've found LLM tech good for stuff that is mostly boilerplate (e.g. some general software development stuff), but when it comes to modeling work that is necessarily bespoke, they are quite bad