Lightnews — Scholar-powered news

David Schlangen @davidschlangen.bsky.social · Jul 20

Bonus post advertising this other thread through the medium of "memes" which I've been told is what you have to do on social media.

Scene from the film "wargames", with an added supertitle saying "How can we evaluate LLMs in interactions? Where do we get the interaction purposes from??", to which Matthew Broderick's character answers: "It's games"

Another still from the film, supertitle "But getting people to play games with the computer takes time!", to which our hero answers "Is there any way to make it play itself?"

The famous scene where the computer say "A strange game. The only winning move is" ... well, it now says, ".. to check out the clembench."

1

David Schlangen @davidschlangen.bsky.social · Jul 20

(That animation in the first post? That's claude trying, and failing, to fully explore a maze in the MapWorld game.)

David Schlangen @davidschlangen.bsky.social · Jul 20

We'd love for other people to use it to test the interaction / agentic abilities of their models, and/or to build new fun and challenging games / interactions!
github.com/clp-research...
github.com/clp-research...
»

GitHub - clp-research/clembench: Collection of games to be run with the clemcore framework

Collection of games to be run with the clemcore framework - clp-research/clembench

github.com

1 1

David Schlangen @davidschlangen.bsky.social · Jul 20

Thanks to a recent short-term grant, we've been able to focus on code quality and ease of use for benchmarking and extensibility. (Exploring new games is a fun programming lab activity, which we've run several times by now!) Here's a writeup of the current state: arxiv.org/abs/2507.08491
»

A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench

There are currently two main paradigms for evaluating large language models (LLMs), reference-based evaluation and preference-based evaluation. The first, carried over from the evaluation of machine l...

arxiv.org

1

David Schlangen @davidschlangen.bsky.social · Jul 20

clembench now spans abstract (e.g., wordle) and concrete tasks (simulated household); language and l+vision; and benchmarking, learning (playpen), and user simulation (clem:todd).
arxiv.org/abs/2504.08590
arxiv.org/abs/2505.05445
»

Playpen: An Environment for Exploring Learning Through Conversational Interaction

Interaction between learner and feedback-giver has come into focus recently for post-training of Large Language Models (LLMs), through the use of reward models that judge the appropriateness of a mode...

arxiv.org

1

David Schlangen @davidschlangen.bsky.social · Jul 20

It's great to see the idea of using games / interactions to evaluate LLMs gain traction, with textarena.ai and now ARC-AGI-3 being latest entrants.
This is something we've been exploring since early 2023 with clembench ( clembench.github.io ), which we've been continuously maintaining & extending. »

1 2

Reposted by David Schlangen

Philipp Mondorf @pmondorf.bsky.social · Jul 18

📄 [ACL 2025 main] LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (doi.org/10.48550/arX...)

LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case...

doi.org

1 4 9

David Schlangen @davidschlangen.bsky.social · May 30

Ha, yes, I'm quite pleased as well with how that turned out. It's nothing fancy, just a nice font, colouring (obviously), fbox, and rotate.

1

David Schlangen @davidschlangen.bsky.social · May 29

This was the outcome of a collaboration that started last year at an ELLIS workshop, and that has brought together many labs (and many master's and PhD students, and PIs).

Much more remains to be explored in "learning in interaction" -- maybe by you?

🤖🧠 #NLP #AI #LLM

2

David Schlangen @davidschlangen.bsky.social · May 29

Oh yes, here's the link to the actual pre-print: arxiv.org/abs/2504.08590

Playpen: An Environment for Exploring Learning Through Conversational Interaction

Interaction between learner and feedback-giver has come into focus recently for post-training of Large Language Models (LLMs), through the use of reward models that judge the appropriateness of a mode...

arxiv.org

1 1 1

David Schlangen @davidschlangen.bsky.social · May 29

We release the framework and the baseline training setups to foster research in the promising new direction of learning in (synthetic) interaction which we believe will provide more effective ways of post-training agentic conversational LLMs. github.com/lm-playpen/p...

GitHub - lm-playpen/playpen: All you need to get started with the LM Playpen Environment for Learning in Interaction.

All you need to get started with the LM Playpen Environment for Learning in Interaction. - lm-playpen/playpen

github.com

1 3

David Schlangen @davidschlangen.bsky.social · May 29

We find that imitation learning through SFT improves performance on unseen game instances, but does not generalise to new games and negatively impacts other skills -- while interactive learning with GRPO shows balanced improvements without loss of skills.

Table 3 from the paper linked in a post below.

1 3

David Schlangen @davidschlangen.bsky.social · May 29

Together with the learning environment, we also define an experimental setup combining gameplay evaluation on unseen games and traditional NLP benchmarks such as MMLU following (Momente’ et al. 2025) arxiv.org/abs/2502.14359

Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests

We examine three evaluation paradigms: standard benchmarks (e.g., MMLU and BBH), interactive games (e.g., Signalling Games or Taboo), and cognitive tests (e.g., for working memory or theory of mind). ...

arxiv.org

1 3

David Schlangen @davidschlangen.bsky.social · May 29

Playpen is a training environment for post-training LLMs through learning in interaction, by self-play of "dialogue games": goal-oriented language-based activities that generate verifiable rewards.

Diagram showing an interaction triangle "interlocutor A -- world -- interlocutor B", except that this is mediated by GM (the "Game Master"), and that A is a learner wrapped around an LLM, and B also is a wrapper around a (non-learning) LLM.

1 2

David Schlangen @davidschlangen.bsky.social · May 29

🚨 New pre-print! (Well, new & much improved version in any case.) 🚨
If you're interested in LLM post-training techniques and in how to make LLMs better "language users", read this thread, introducing the "LM Playpen".

3 5 14

David Schlangen @davidschlangen.bsky.social · May 21

The University of Potsdam invites applications for 5 postdoc positions, incl. Cognitive Sciences, incl. NLP (esp. cognitive).

These are fairly independent research positions that will allow the candidate to build their own profile. Dln June 2nd.

Details: tinyurl.com/pd-potsdam-2...

#NLProc #AI 🤖🧠

tinyurl.com

2 2

David Schlangen @davidschlangen.bsky.social · May 14

There's indeed suddenly a bit of flexibility in a system that's not exactly known for that.. If there's anyone (post-doc, tenure-track, or more senior) in the #NLP space currently in the US who'd like to explore possiblities in Potsdam, contact me.

🤖🧠

www.nytimes.com/2025/05/14/b...

The World Is Wooing U.S. Researchers Shunned by Trump

www.nytimes.com

1

David Schlangen @davidschlangen.bsky.social · May 7

"We ablated both algorithm and hyperparameter choices [...]"

When did "to ablate" take on the meaning "to systematically vary"? I've noticed this only recently, but it's seems to be super common now.

1 2

Reposted by David Schlangen

David Schlangen @davidschlangen.bsky.social · Apr 15

Update 2: New pre-print! Outcome of an ELLIS workshop last year, & more than a year of discussions and work, across labs and countries: Meet the Playpen, an environment for exploring learning in dialogic interaction.

arxiv.org/abs/2504.08590

1/2

Titlepage of the paper linked in the post.

1 1 4

David Schlangen @davidschlangen.bsky.social · Apr 15

[Sneak preview: If you're wondering where this is going, have a secret look at lm-playschool.github.io -- and stay tuned for more info!]

3/2

A Playschool for LLMs

lm-playschool.github.io

David Schlangen @davidschlangen.bsky.social · Apr 15

Nice baseline results as well: learning via SFT from transcripts does a bit, but only "real"(-ish) learning in interaction (GRPO) generalises. (Basically, you want to see the whole row being green in this table.)

2/2

1 1

David Schlangen @davidschlangen.bsky.social · Apr 15

Update 2: New pre-print! Outcome of an ELLIS workshop last year, & more than a year of discussions and work, across labs and countries: Meet the Playpen, an environment for exploring learning in dialogic interaction.

arxiv.org/abs/2504.08590

1/2

1 1 4

David Schlangen @davidschlangen.bsky.social · Apr 15

This is only a subset of the models on the leaderboard, visit the site to see all 32 models, and also the results for the multimodal version of the benchmark.

David Schlangen @davidschlangen.bsky.social · Apr 15

Update 1: New models added to our dialogue game-based agentic LLM leaderboard. TL;DR: GPT-4.1 as good as 4o, but much cheaper. Llama4 indeed not very good (decisively worse than 3.2 70B!). OLMo decent, but there's still a secret sauce that only closed labs have.

clembench.github.io

Screenshot of leaderboard as linked in post.

1 1

Reposted by David Schlangen

arxiv cs.CL @arxiv-cs-cl.bsky.social · Apr 14

Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Moment\`e, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, ...
Playpen: An Environment for Exploring Learning Through Conversational Interaction
https://arxiv.org/abs/2504.08590

2 1