Lightnews — Scholar-powered news

boydgraber.bsky.social @boydgraber.bsky.social · 21d

We had come at it more from the position of trying to use as few dev examples as possible (to keep them secret). I.e., use the best items you could and every model uses the exact same. But it makes sense to use the adaptive testing scenario if you don't mind potentially exposing more dev.

1 1

boydgraber.bsky.social @boydgraber.bsky.social · 21d

Link to paper since I ran out of room:

users.umiacs.umd.edu/~ying/docs/2...

users.umiacs.umd.edu

1 1

boydgraber.bsky.social @boydgraber.bsky.social · 21d

In 2021, we proposed using IRT to find bad examples and to create more targeted leaderboards (Evaluation
Examples Are Not Equally Informative: How Should That Change NLP Leaderboards?).

From my reading, the big difference seems to be that they're also using the agent's skill, which is super cool!

1 2

boydgraber.bsky.social @boydgraber.bsky.social · 21d

We also found that it's helpful for improving uncertainty estimation of models:

arxiv.org/abs/2205.12507

arxiv.org

boydgraber.bsky.social @boydgraber.bsky.social · Sep 2

If it said that 1990 was "about 10 years ago", I would say that it has reached tenured faculty-level intelligence.

1

boydgraber.bsky.social @boydgraber.bsky.social · Aug 22

Today's the deadline to apply for an AI-specific teaching track position at UMD:

umd.wd1.myworkdayjobs.com/UMCP/job/Uni...

Please join us!

2

boydgraber.bsky.social @boydgraber.bsky.social · Aug 21

A couple of weeks ago I left my family behind at a cable car station to finish climbing to the peak of a mountain because they were too scared to continue. When I reached the top, my phone gave a notification: new podcasts available for download. Apparently LMU has an observatory on Wendelstein.

3

boydgraber.bsky.social @boydgraber.bsky.social · Aug 4

Do you mean salary, physical facilities, work environment, or funding ecosystem?

1

boydgraber.bsky.social @boydgraber.bsky.social · Jul 28

At the risk of picking out one of my favorite children, this was the paper with our best traditional video of this cycle (thanks to Jon May for playing along):

t.co/QQlgwzo6jf
t.co/2G6kwAAPMy

https://youtu.be/L_hcHQep3fc

t.co

boydgraber.bsky.social @boydgraber.bsky.social · Jul 28

Finally, this evening I'll be standing in for
@wwongkamjan.bsky.social
at the Findings poster (18:00, Hall X4/X5): Should I Trust You? Detecting Deception in Negotiations using Counterfactual RL

x.com/joywwong/sta...

Joy Wongkamjan on X: "Our paper CTRL-D is accepted to ACL Findings and will be presented at ACL 2025! 🗓️Poster session: 18:00–19:30 (Level 0 Exhibit Halls X4/X5) I’m sad I can’t be there, but Jordan (@boydgraber) will! You’ll enjoy learning about CTRL-D from him. Now… what is CTRL-D? 🔍 https://t.co/ucIPZRHBF1" / X

Our paper CTRL-D is accepted to ACL Findings and will be presented at ACL 2025! 🗓️Poster session: 18:00–19:30 (Level 0 Exhibit Halls X4/X5) I’m sad I can’t be there, but Jordan (@boydgraber) will! You’ll enjoy learning about CTRL-D from him. Now… what is CTRL-D? 🔍 https://t.co/ucIPZRHBF1

x.com

1

boydgraber.bsky.social @boydgraber.bsky.social · Jul 28

t.co/j3Iibs9hEn
www.youtube.com/watch?v=NJKd...

https://www.cs.umd.edu/~jbg//docs/2025_acl_grace.pdf

t.co

1

boydgraber.bsky.social @boydgraber.bsky.social · Jul 28

In the second oral paper (14:22 PM, Room 1.62),
@yysung.bsky.social is presenting: GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration

x.com/YooYeonSung1...

(Short version: quiz bowl, a dumb trivia game, shows humans' calibration > LLMs'.)

Yoo Yeon Sung@ACL2025 on X: "I’ll be presenting this work in Room 1.62 today! If you're curious about how calibration errors in LLMs can be measured through human calibration, come find me and @enfleisig! 📍Oral Session 3 - HC: Human-centered NLP 📅Monday, July 28@ 2PM" / X

I’ll be presenting this work in Room 1.62 today! If you're curious about how calibration errors in LLMs can be measured through human calibration, come find me and @enfleisig! 📍Oral Session 3 - HC: Human-centered NLP 📅Monday, July 28@ 2PM

x.com

1

boydgraber.bsky.social @boydgraber.bsky.social · Jul 28

t.co/LagmrMjVgi
t.co/aGM7LC2m0q

https://youtu.be/wuEIeydhamA

t.co

1

boydgraber.bsky.social @boydgraber.bsky.social · Jul 28

In the first poster session (11AM Monday, Hall X4/X5),
@nbalepur.bsky.social is presenting: Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas

x.com/NishantBalep...

Nishant is ill-prepared for ACL2025 on X: "While personalization is great, it's not perfect. We found our strategy could easily let users jailbreak the model 😭 With some extra safeguards (e.g. refusal training), we think inferred personas could become a promising way to boost personalization in post-training recipes! 🧑‍🍳 https://t.co/Fpu1Bl34fI" / X

While personalization is great, it's not perfect. We found our strategy could easily let users jailbreak the model 😭 With some extra safeguards (e.g. refusal training), we think inferred personas could become a promising way to boost personalization in post-training recipes! 🧑‍🍳 https://t.co/Fpu1Bl34fI

x.com

1

boydgraber.bsky.social @boydgraber.bsky.social · Jul 28

My students and I are presenting three papers on Monday at #ACL2025 and this thread will recap them (including their videos).

1 2 7

boydgraber.bsky.social @boydgraber.bsky.social · Jul 18

The precursor to this paper "The Incoherence of Coherence" had our most-watched paper video ever, so I thought we had to surpass it somehow ... so we decided to do a song parody (of Roxanne, obviously):

youtu.be/87OBxEM8a9E

2 7

boydgraber.bsky.social @boydgraber.bsky.social · Jul 14

Which makes this:
users.umiacs.umd.edu/~ying/docs/n...

"The Hobbit"

users.umiacs.umd.edu

2

boydgraber.bsky.social @boydgraber.bsky.social · Jun 17

And you can signup for online mirror (June 21, 12:00 EST) here:
docs.google.com/forms/d/e/1F...

[Signup deadline: June 18 Anywhere on Earth]

2025 QANTA Player Signup

Sign up for the human competition for our 2025 QANTA event. More information: https://sites.google.com/view/qanta/2025-competition/2025-human-teams

docs.google.com

boydgraber.bsky.social @boydgraber.bsky.social · Jun 17

If this sounds like fun, we have more information on the setup here:
sites.google.com/view/qanta/2...

QANTA Project - 2025 Human Teams

How this Works It's always difficult to try something new, but we think this will be fun! Short Version: You'll play multiple rounds of tossup-bonus quiz bowl with computer teammates. The computer ca...

sites.google.com

1

boydgraber.bsky.social @boydgraber.bsky.social · Jun 17

There are more reflections here:
hsquizbowl.org/forums/viewt...

Human-Computer Cooperation Tournament (College Park: June 14, Online: June 21) - The Quizbowl Resource Center

Sponsored by the Partnership for Academic Competition Excellence

hsquizbowl.org

1

boydgraber.bsky.social @boydgraber.bsky.social · Jun 17

Sara’s Crias went 4-2 to win the tournament (and $150 dollars). Noah Sheidlower’s music packet was the most difficult for computers, and Jame Carlson’s Spatial Reasoning was the fan favorite. We’ll announce writer and computer prizes after our online mirror. (And also post the packets.)

1

boydgraber.bsky.social @boydgraber.bsky.social · Jun 17

We had our first human–computer cooperative AI tournament at the UMD. Key takeaways: 1) computers are getting better at trivia 2) they still suck at calibration 3) our teaming mechanic kept the games competitive and mostly fun (at least that’s what the players said).

Human-Computer AI Collaborative Tournament Gameplay

1

boydgraber.bsky.social @boydgraber.bsky.social · Jun 10

And get more information here:
sites.google.com/view/qanta/2...

QANTA Project - 2025 Human Teams

How this Works It's always difficult to try something new, but we think this will be fun! Short Version: You'll play multiple rounds of tossup-bonus quiz bowl with computer teammates. The computer ca...

sites.google.com

boydgraber.bsky.social @boydgraber.bsky.social · Jun 10

Sign up to play here:
docs.google.com/forms/d/e/1F...

2025 QANTA Player Signup

Sign up for the in-person competition for our 2025 QANTA event. More information: https://sites.google.com/view/qanta/2025-competition/2025-human-teams

docs.google.com

1

boydgraber.bsky.social @boydgraber.bsky.social · Jun 10

Today is the deadline to sign up for our Human-Computer trivia competition held on June 14, 2024 in College Park, MD. $150 prize for the team who can answer the most questions with the help of an AI.

1 1