One of the most exciting results: For math problems, reasoning chains with a higher PHi loss are significantly more likely to be correct. The model essentially signals when it's "working hard" to find the right answer.

And it works! PHi Loss cleanly separates "interesting" tasks (like in-context learning, modeling new code/literature) from "boring" ones (memorization, random data), while next-token loss doesn't. We can use pre-trained LLMs or models trained from scratch.

Our solution: Instead of next token loss, we measure the predictability of the model's hidden state.
We introduce the PHi (Prediction of Hidden states) layer and PHi Loss. High PHi loss means the model's hidden state is complex and unpredictable—a sign of interesting computation.

#ICML paper with co-authors @robertcsordas.bsky.social and Jürgen Schmidhuber, at poster session east this afternoon:)

How can we tell if an LLM is actually "thinking" versus just spitting out memorized or trivial text? Can we detect when a model is doing anything interesting?

(Thread below👇)