Lightnews — Scholar-powered news

LightNews

Kenny Peng

@kennypeng.bsky.social

100 followers 180 following 30 posts

CS PhD student at Cornell Tech. Interested in interactions between algorithms and society. Princeton math '22. kennypeng.me

Posts Media Videos Starter Packs

Kenny Peng @kennypeng.bsky.social · Aug 12

"those already relatively advantaged are, empirically, more able to pay time costs and navigate administrative burdens imposed by the mechanisms."

This point by @nkgarg.bsky.social has greatly shaped my thinking about the role of computer science in public service settings.

Nikhil Garg @nkgarg.bsky.social · Aug 11

New piece, out in the Sigecom Exchanges! It's my first solo-author piece, and the closest thing I've written to being my "manifesto." #econsky #ecsky
arxiv.org/abs/2507.03600

Screenshot of paper abstract, with text: "A core ethos of the Economics and Computation (EconCS) community is that people have complex private preferences and information of which the central planner is unaware, but which an appropriately designed mechanism can uncover to improve collective decisionmaking. This ethos underlies the community’s largest deployed success stories, from stable matching systems to participatory budgeting. I ask: is this choice and information aggregation “worth it”? In particular, I discuss how such systems induce heterogeneous participation: those already relatively advantaged are, empirically, more able to pay time costs and navigate administrative burdens imposed by the mechanisms. I draw on three case studies, including my own work – complex democratic mechanisms, resident crowdsourcing, and school matching. I end with lessons for practice and research, challenging the community to help reduce participation heterogeneity and design and deploy mechanisms that meet a “best of both worlds” north star: use preferences and information from those who choose to participate, but provide a “sufficient” quality of service to those who do not."

Kenny Peng @kennypeng.bsky.social · Aug 5

How do we reconcile excitement about sparse autoencoders with negative results showing that they underperform simple baselines? Our new position paper makes a distinction: SAEs are very useful for tools for discovering *unknown* concepts, less good for acting on *known* concepts.

Kenny Peng @kennypeng.bsky.social · Jul 30

One paragraph pitch for why sparse autoencoders are cool (they learn *interpretable* text embeddings)

$Text embeddings capture tons of information, but individual dimensions are uninterpretable. It would be great if each dimension reflected a concept (“dimension 12 is about cats”). But text embeddings are ~1000 dimensions and there are millions of human concepts. So we need a higher dimensional embedding. Now notice that while there are tons of human concepts, they appear *sparsely*—any piece of text invokes a tiny fraction of concepts. This motivates learning a sparse high-dimensional encoding of text embeddings. Turns out SAEs work great for this in practice, producing *interpretable text embeddings*.$

Kenny Peng @kennypeng.bsky.social · Jul 16

This is collectively joint work with @rajmovva.bsky.social, @nkgarg.bsky.social, Jon Kleinberg @emmapierson.bsky.social, Elliot Kim, and Avi Garg.

Come chat with us!

Kenny Peng @kennypeng.bsky.social · Jul 16

In Correlated Errors in Large Language Models, we show that LLMs are correlated in how they make mistakes. On one dataset, LLMs make the same mistake 2x more than random chance.

Kenny Peng @kennypeng.bsky.social · Jul 16

In Sparse Autoencoders for Hypothesis Generation, we show that SAEs can be used to find predictive natural language concepts in text data (e.g., that "addresses collective human responsibility" predicts lower headline engagement), achieving SOTA performance and efficiency.

Kenny Peng @kennypeng.bsky.social · Jul 16

We're presenting two papers Wednesday at #ICML2025, both at 11am.

Come chat about "Sparse Autoencoders for Hypothesis Generation" (west-421), and "Correlated Errors in LLMs" (east-1102)!

Short thread ⬇️

Kenny Peng @kennypeng.bsky.social · Jul 3

This is joint work with Elliot Kim and Avi Garg (co-leads), and @nkgarg.bsky.social. We’ll be presenting at ICML in two weeks! Come see us at East Exhibition Hall A-B #E-2905 (11am-1:30pm Wednesday). (7/7)

Kenny Peng @kennypeng.bsky.social · Jul 3

The empirics here also add nuance to theory. While less correlated models can be more accurate together through a “wisdom of crowds,” this effect doesn't hold when newer, more correlated models are adopted in our simulations: gains from individual accuracy outweigh losses from homogeneity. (6/7)

A chart showing the assortativity of different markets. The market with the newest models is the most assortative.

Kenny Peng @kennypeng.bsky.social · Jul 3

However, in equilibrium, increased correlation—as predicted by past theoretical work—actually improves average applicant outcomes (intuitively, of the applicants who receive a job offer, more correlation means they will get more offers).

For the theory, see arxiv.org/abs/2312.09841 (5/7)

A chart showing that increased homogeneity on models leads to better average applicant outcomes.

Kenny Peng @kennypeng.bsky.social · Jul 3

Since LLMs are correlated, this also leads to greater systemic exclusion in a labor market setting: more applicants are screened out of all jobs. Systemic exclusion persists even when different LLMs are used across firms. (4/7)

A chart showing that the systemic exclusion rate remains significant even when many different LLMs are used in a resume screening task.

Kenny Peng @kennypeng.bsky.social · Jul 3

A consequence of error correlation is that LLM judges inflate accuracy of models less accurate than it. Here, we plot accuracy inflation against true model accuracy. Models from the same company (in red) are especially inflated. (3/7)

Three charts side by side that show that different judge models inflate the accuracy of other models.

Kenny Peng @kennypeng.bsky.social · Jul 3

What explains error correlation? We found that models from the same company are more correlated. Strikingly, more accurate models also have more correlated errors, suggesting some level of convergence among newer models. (2/7)

A regression table showing that models from the same company, with the same architecture, and that are more accurate tend to have more correlated errors.

Kenny Peng @kennypeng.bsky.social · Jul 3

Are LLMs correlated when they make mistakes? In our new ICML paper, we answer this question using responses of >350 LLMs. We find substantial correlation. On one dataset, LLMs agree on the wrong answer ~2x more than they would at random. 🧵(1/7)

arxiv.org/abs/2506.07962

Heat map showing that more accurate models have more correlated errors.

Reposted by Kenny Peng

Divya Shanmugam @dmshanmugam.bsky.social · Jun 14

New work 🎉: conformal classifiers return sets of classes for each example, with a probabilistic guarantee the true class is included. But these sets can be too large to be useful.

In our #CVPR2025 paper, we propose a method to make them more compact without sacrificing coverage.

A gif explaining the value of test-time augmentation to conformal classification. The video begins with an illustration of TTA reducing the size of the predicted set of classes for a dog image, and goes on to explain that this is because TTA promotes the true class's predicted probability to be higher, even when it's predicted to be unlikely.

Reposted by Kenny Peng

Raj Movva @rajmovva.bsky.social · May 5

We'll present HypotheSAEs at ICML this summer! 🎉
Draft: arxiv.org/abs/2502.04382

We're continuing to cook up new updates for our Python package: github.com/rmovva/Hypot...

(Recently, "Matryoshka SAEs", which help extract coarse and granular concepts without as much hyperparameter fiddling.)

Reposted by Kenny Peng

Erica Chiang @ericachiang.bsky.social · May 1

I’m really excited to share the first paper of my PhD, “Learning Disease Progression Models That Capture Health Disparities” (accepted at #CHIL2025)! ✨ 1/

📄: arxiv.org/abs/2412.16406

Reposted by Kenny Peng

Emma Pierson @emmapierson.bsky.social · Apr 25

The US government recently flagged my scientific grant in its "woke DEI database". Many people have asked me what I will do.

My answer today in Nature.

We will not be cowed. We will keep using AI to build a fairer, healthier world.

www.nature.com/articles/d41...

My ‘woke DEI’ grant has been flagged for scrutiny. Where do I go from here?

My work in making artificial intelligence fair has been noticed by US officials intent on ending ‘class warfare propaganda’.

Kenny Peng @kennypeng.bsky.social · Apr 2

This was lots of fun. Dogathoners: @gsagostini.bsky.social @sidhikabalachandar.bsky.social @ericachiang.bsky.social @nkgarg.bsky.social @laufer.bsky.social @rajmovva.bsky.social @emmapierson.bsky.social @shuvoms.bsky.social @dmshanmugam.bsky.social

Data: data.cityofnewyork.us/Health/NYC-D...

NYC Dog Licensing Dataset | NYC Open Data

data.cityofnewyork.us

Kenny Peng @kennypeng.bsky.social · Apr 2

We also made a web app (github.com/shuvom-s/nyc...) you can run locally to make maps by name/breed.

We also briefly explored a “dog park simulator.”

Kenny Peng @kennypeng.bsky.social · Apr 2

7) Dogs are licensed the most in July and the least in November.

Kenny Peng @kennypeng.bsky.social · Apr 2

6) Dog names are shorter than baby names; 30% of dogs have names less than 5 letters long, but only 20% of babies.

Kenny Peng @kennypeng.bsky.social · Apr 2

5) Goldendoodles, Poodle Crossbreeds, and French Bulldogs are becoming more popular. Chihuahua's are becoming less popular.

Luna is the only popular name that is becoming more common among dogs AND babies.

Kenny Peng @kennypeng.bsky.social · Apr 2

4) The “most common dog” in NYC is a Yorkshire Terrier named Bella. Jack Russel Terriers are often “Jack” and Charles Spaniels “Charlie.” Huskies are always named Luna—the reason for which is unclear (?).

Kenny Peng @kennypeng.bsky.social · Apr 2

3) There are hundreds of categories of dog names (which we found using a sparse autoencoder): math/mathematical concepts, pasta varieties, and “sudden or energetic movement or force” to name a few

kennypeng.me/nycdogs/inde...

DOG NAME CATEGORIES