Lightnews — Scholar-powered news

Cas (Stephen Casper)

@scasper.bsky.social

140 followers 180 following 110 posts

AI technical gov & risk management research. PhD student @MIT_CSAIL, fmr. UK AISI. I'm on the CS faculty job market! https://stephencasper.com/

stephencasper.com

Posts Media Videos Starter Packs

Pinned

Cas (Stephen Casper) @scasper.bsky.social · Sep 4

📌📌📌
I'm excited to be on the faculty job market this fall. I just updated my website with my CV.
stephencasper.com

Stephen Casper

Visit the post for more.

stephencasper.com

4 17

Cas (Stephen Casper) @scasper.bsky.social · 6d

Don't forget that in AI, "sycophancy," "pandering," "personalized alignment," "steerable alignment," and "user alignment" all describe exactly the same thing.

1 4

Cas (Stephen Casper) @scasper.bsky.social · 7d

Almost 2 years out from my paper with Carson Ezell et al. titled "Black-Box Access is Insufficient for Rigorous AI Audits," it's cool to see that AI companies are starting to report on [internal] evals that use fine-tuning or interp-based methods.

Cas (Stephen Casper) @scasper.bsky.social · 9d

A monumental milestone for AI governance:

www.gov.ca.gov/2025/09/29/g...

1 6

Reposted by Cas (Stephen Casper)

Jacy Reese Anthis @jacyanthis.bsky.social · 23d

LLM agents are optimized for thumbs-up instant gratification. RLHF -> sycophancy

We propose human agency as a new alignment target in HumanAgencyBench, made possible by AI simulation/evals. We find e.g., Claude most supports agency but also most tries to steer user values 👇 arxiv.org/abs/2509.08494

The main figure from the HumanAgencyBench paper, showing five models across the six dimensions. The table of results in the appendix has this information too.

2 2 14

Cas (Stephen Casper) @scasper.bsky.social · Sep 8

I'll be leading a MATS stream this winter with a focus on technical AI governance. You can apply here by October 2!

www.matsprogram.org/apply

Apply for Winter 2026 — ML Alignment & Theory Scholars

www.matsprogram.org

Cas (Stephen Casper) @scasper.bsky.social · Sep 4

📌📌📌
I'm excited to be on the faculty job market this fall. I just updated my website with my CV.
stephencasper.com

Stephen Casper

Visit the post for more.

stephencasper.com

4 17

Cas (Stephen Casper) @scasper.bsky.social · Aug 30

✅

Cas (Stephen Casper) @scasper.bsky.social · Aug 30

Here is a riddle I came up with for a draft to illustrate the differences between normal chat models and reasoning models. Can you figure it out?

Dark as night in the morning light.
I live high until I am ground.
I sit dry until I am drowned.
What am I?

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

...But in a case like this, I think our challenge is one about scalable oversight and maybe about performing RL-finetuning.

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Suppose we have a system performing very hard and complex tasks that we don't know how to evaluate. I agree with the concern about knowing whether or not some sort of testing setup is evaluating the system to its full potential...

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Thoughts?

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Question 3: Does few-shot fine-tuning on the test task (or some related task) beat the method being studied? If so, why is it worth studying?

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Question 2: If this work is related to a model's "intentions," what are those, and why does it matter?

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Question 1: Why call it "sandbagging" and not "capability elicitation" or "eval gaming"?

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

So at a minimum, I think that anyone working on "sandbagging" should have clear answers to a few questions:

1 2

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Here are some papers I sometimes cite to make this point.

arxiv.org/abs/2502.05209
arxiv.org/abs/2405.19550
arxiv.org/abs/2502.02180
arxiv.org/abs/2407.07890
arxiv.org/abs/2412.07097

On Evaluating the Durability of Safeguards for Open-Weight LLMs

Stakeholders -- from model developers to policymakers -- seek to minimize the dual-use risks of large language models (LLMs). An open challenge to this goal is whether technical safeguards can...

arxiv.org

1 3

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Second, I think "sandbagging" is probably already a solved problem. Mounting evidence suggests that, if a model has a capability--even one that is adversarially hidden--few-shot fine-tuning on the target task can elicit it.

1 3

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Pontificating about a system's 'intentions' doesn't shed any light on the technical problem of eliciting its capabilities. It just confuses people in a characteristically AI-safety-community way.

1 3

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

...framing "sandbagging" as (largely) a problem about strategic gaming from a model defines it (largely) in terms of a model's intentions. But there's no pretense, precision, or point to this.

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

And that's the first reason I am cold on "sandbagging." It reinvents and renames the goals of capability elicitation and rigorous algorithmic audits. Both already have much existing research.

But the sequel seems worse...

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

"Sandbagging" is defined as "strategic underperformance on an evaluation," whether by a model or developer. In other words, "sandbagging" just means that an evaluation didn't successfully elicit a system's full capabilities.

1 1

Cas (Stephen Casper) @scasper.bsky.social · Aug 24

Research on AI "sandbagging" is getting more popular recently. In this 🧵, I'll give some reasons that I think it's not a useful research paradigm.

TL;DR, I think it's a confusing reframing of fairly well studied and previously solved problems.

1 7

Cas (Stephen Casper) @scasper.bsky.social · Aug 23

There have been a couple cool pieces up recently debunking the "China is racing on AI, so the US must too" narrative.

time.com/7308857/chin...

papers.ssrn.com/sol3/papers....

1 4

Cas (Stephen Casper) @scasper.bsky.social · Aug 22

A personal update:
- I just finished my 6-month residency at UK AISI.
- I'm going back to MIT for the final year of my PhD.
- I'm on the postdoc and faculty job markets this fall!

Cas (Stephen Casper) @scasper.bsky.social · Aug 18

...Currently, there's just very little open research available on the nuanced effects--intended and unintended--of data curation on model capabilities. So my reaction is to emphasize the value of more open reporting and research on this kind of stuff.