Lightnews — Scholar-powered news

Reposted by Kevin Feng

Alicia Guo @aliciaguo.com · 29d

earlier this summer I published my first paper of my phd! ✨ a qualitative study on how creative writers are using AI in their writing and what their strategies were in order to align with their personal writing values

1 4 17

Kevin Feng @kjfeng.me · Aug 1

Special thanks to @katygb.bsky.social and @sethlazar.org for organizing a fantastic @knightcolumbia essay series on AI and Democratic freedoms, and other authors of the essay series for valuable discussion + feedback. Check out the other essays here! knightcolumbia.org/research/art....

Artificial Intelligence and Democratic Freedoms

knightcolumbia.org

1

Kevin Feng @kjfeng.me · Aug 1

Finally, autonomy levels can help with agentic safety evaluations and setting thresholds for autonomy risks in safety frameworks. How do we know what level an agent is operating at, and what appropriate risk mitigations should be put in place? Read our paper for more!

1 2

Kevin Feng @kjfeng.me · Aug 1

We also introduce **agent autonomy certificates** to govern agent behavior in single- and multi-agent settings. Certificates constrain the space of permitted actions and help ensure effective and safe collaborative behaviors in agentic systems.

1 1

Kevin Feng @kjfeng.me · Aug 1

Why have this framework at all?

We argue that agent autonomy can be a deliberate design decision, independent of the agent's capability or environment. This framework offers a practical guide for designing human-agent & agent-agent collaboration for real-world deployments.

1

Kevin Feng @kjfeng.me · Aug 1

Level 5: user as an observer.

The agent autonomously operates over long time horizons and does not seek user involvement at all. The user passively monitors the agent via activity logs and has access to an emergency off switch.

Example: Sakana AI's AI Scientist.

1

Kevin Feng @kjfeng.me · Aug 1

Level 4: user as an approver.

The agent only requests user involvement when it needs approval for a high-risk action (e.g., writing to a database) or when it fails and needs user assistance.

Example: most coding agents (Cursor, Devin, GH Copilot Agent, etc.).

1

Kevin Feng @kjfeng.me · Aug 1

Level 3: user as a consultant.

The agent takes the lead in task planning and execution, but actively consults the user to elicit rich preferences and feedback. Unlike L1 & L2, the user can no longer directly control the agent's workflow.

Example: deep research systems.

1

Kevin Feng @kjfeng.me · Aug 1

Level 2: user as a collaborator.

The user and the agent collaboratively plan and execute tasks, handing off information to each other and leveraging shared environments and representations to create common ground.

Example: Cocoa (arxiv.org/abs/2412.10999).

1

Kevin Feng @kjfeng.me · Aug 1

Level 1: user as an operator.

The user is in charge of high-level planning to steer the agent. The agent acts when directed, providing on-demand assistance.

Example: your average "copilot" that drafts your emails when you ask it to.

1

Kevin Feng @kjfeng.me · Aug 1

Web: knightcolumbia.org/content/leve....
arXiv: arxiv.org/abs/2506.12469.

Co-authored w/ David McDonald and @axz.bsky.social .

Summary thread below.

Levels of Autonomy for AI Agents

knightcolumbia.org

1

Kevin Feng @kjfeng.me · Aug 1

Here are links to the paper:

Web: knightcolumbia.org/content/leve....
arXiv: arxiv.org/abs/2506.12469.

Co-authored w/ David McDonald and @axz.bsky.social .

Levels of Autonomy for AI Agents

knightcolumbia.org

Kevin Feng @kjfeng.me · Aug 1

Special thanks to @katygb.bsky.social and @sethlazar.org for organizing a fantastic @knightcolumbia.org essay series on AI and Democratic freedoms, and other authors of the essay series for valuable discussion + feedback. Check out the other essays here! knightcolumbia.org/research/art....

Artificial Intelligence and Democratic Freedoms

knightcolumbia.org

1

Kevin Feng @kjfeng.me · Aug 1

Finally, autonomy levels can help with agentic safety evaluations and setting thresholds for autonomy risks in safety frameworks. How do we know what level an agent is operating at, and what appropriate risk mitigations should be put in place? Read our paper for more!

1

Kevin Feng @kjfeng.me · Aug 1

We also introduce **agent autonomy certificates** to govern agent behavior in single- and multi-agent settings. Certificates constrain the space of permitted actions and help ensure effective and safe collaborative behaviors in agentic systems.

1

Kevin Feng @kjfeng.me · Aug 1

Why have this framework at all?

We argue that agent autonomy can be a deliberate design decision, independent of the agent's capability or environment. This framework offers a practical guide for designing human-agent & agent-agent collaboration for real-world deployments.

1

Kevin Feng @kjfeng.me · Aug 1

Level 5: user as an observer.

The agent autonomously operates over long time horizons and does not seek user involvement at all. The user passively monitors the agent via activity logs and has access to an emergency off switch.

Example: Sakana AI's AI Scientist.

1

Kevin Feng @kjfeng.me · Aug 1

Level 4: user as an approver.

The agent only requests user involvement when it needs approval for a high-risk action (e.g., writing to a database) or when it fails and needs user assistance.

Example: most coding agents (Cursor, Devin, GH Copilot Agent, etc.).

1

Kevin Feng @kjfeng.me · Aug 1

Level 3: user as a consultant.

The agent takes the lead in task planning and execution, but actively consults the user to elicit rich preferences and feedback. Unlike L1 & L2, the user can no longer directly control the agent's workflow.

Example: deep research systems.

1

Kevin Feng @kjfeng.me · Aug 1

Level 2: user as a collaborator.

The user and the agent collaboratively plan and execute tasks, handing off information to each other and leveraging shared environments and representations to create common ground.

Example: Cocoa (arxiv.org/abs/2412.10999).

1

Kevin Feng @kjfeng.me · Aug 1

Level 1: user as an operator.

The user is in charge of high-level planning to steer the agent. The agent acts when directed, providing on-demand assistance.

Example: your average "copilot" that drafts your emails when you ask it to.

1

Kevin Feng @kjfeng.me · Aug 1

📢 New paper, published by @knightcolumbia.org

We often talk about AI agents augmenting vs. automating work, but how exactly can different configurations of human-agent interaction look like? We introduce a 5-level framework for AI agent autonomy to unpack this.

🧵👇

Screenshot of a paper on a webpage, with a figure showing 5 levels of autonomy for AI agents and the tradeoff between user involvement and agent autonomy as the levels increase.

1 5 21

Kevin Feng @kjfeng.me · Aug 1

Hi Dave, this is super cool! I'm currently working on a research prototype that brings domain experts together to draft policies for AI behavior and would love to chat more, esp about how I can integrate something like this

2

Reposted by Kevin Feng

Knight First Amendment Institute @knightcolumbia.org · Jul 28

In an essay for our AI & Democratic Freedoms series,
@kjfeng.me, @axz.bsky.social (both of @socialfutureslab.bsky.social), & David W. McDonald outline a framework for levels of #AI agent autonomy that can be used to support the responsible deployment of AI agents. knightcolumbia.org/content/leve...

Levels of Autonomy for AI Agents

knightcolumbia.org

3 3

Kevin Feng @kjfeng.me · Apr 27

Excited to kick off the first workshop on Sociotechnical AI Governance at #chi2025 (STAIG@CHI'25) with a morning poster session in a full house! Looking forward to more posters, discussions, and our keynote in the afternoon. Follow our schedule at chi-staig.github.io!

1 3