👥 @veraneplenbroek.bsky.social, Sandro Pezelle, @barbaraplank.bsky.social, @davidschlangen.bsky.social, Alessandro Suglia, @akskuchi.bsky.social, @ecekt.bsky.social, and @alberto-testoni.bsky.social.
📍Poster Session 2 — Hall 4/5, 11:00–12:30, Monday, July 28.

#MaiNLP #MCML #NLProc

👥 Special thanks to @annabavaresco.bsky.social, @raffagbernardi.bsky.social, @leobertolazzi.bsky.social, @delliott.bsky.social, Raquel Fernández, Albert Gatt, @esamghaleb.bsky.social, Mario Giulianelli, @michaelwhanna.bsky.social, @akoller.bsky.social, @andre-t-martins.bsky.social

👥 This work is the result of a wonderful collaboration involving 20 researchers from 11 different universities.

🔎Based on evaluations across 11 recent LLMs, we find that model judgments should be used with care, as they exhibit notable variability depending on the task and samples being evaluated. We argue that LLMs should be carefully validated against human judgments before being used as evaluators.

🔎 In this work, we study whether LLM judgments can be reliably used as proxies for human judgments. We introduce JUDGE-BENCH, an extensive collection of 20 datasets with human annotations covering a variety of NLP tasks.

📄 [ACL 2025 main] LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (doi.org/10.48550/arX...)

👥 Huge thanks to my collaborators and co-authors, Sondre Wold and @barbaraplank.bsky.social
📍Poster Session 7 — Hall 4/5, 10:30–12:00, Tuesday, July 29.

🔎 Moreover, we show that these circuits can be reused and combined through set operations to represent more complex functional capabilities of the model. For more information, check out the paper!

🔎 In this work, we study the relationship between transformer circuits identified for highly compositional and functionally related tasks. We find that functionally similar circuits exhibit both notable node overlap and cross-task faithfulness.

📄 [ACL 2025 main] Circuit compositions: Exploring Modular Structures in Transformer-Based Language Models (doi.org/10.48550/arX...)

I am happy to share that I’ll be attending #ACL2025 in Vienna 🇦🇹, where I’ll be presenting two papers (more information below)!

🎉MaiNLP is turning 3 today!🎂🥳 We’ve grown a lot since @barbaraplank.bsky.social started this group with nothing but three aspiring researches and a hand-drawn sign on the door. Huge thanks to all the amazing people who have joined or visited us since. Here’s to many more years of exciting research!🚀

🙋‍♂️