Ryan Steed
@rbsteed.com
AI Policy Fellow @ Princeton | PhD Carnegie Mellon | privacy, accountability, & algorithmic systems
Pinned
Ryan Steed
@rbsteed.com
· 3d
Accelerating AI Innovation Through Measurement Science
Building gold-standard AI systems requires gold-standard AI measurement science – the scientific study of methods used to assess AI systems’ properties and impacts. NIST works to improve measurements ...
www.nist.gov
Our team at NIST's Center for AI Standards and Innovation (CAISI) just released a blog post with open questions for AI measurement science:
www.nist.gov/blogs/caisi-...
www.nist.gov/blogs/caisi-...
Our team at NIST's Center for AI Standards and Innovation (CAISI) just released a blog post with open questions for AI measurement science:
www.nist.gov/blogs/caisi-...
www.nist.gov/blogs/caisi-...
Accelerating AI Innovation Through Measurement Science
Building gold-standard AI systems requires gold-standard AI measurement science – the scientific study of methods used to assess AI systems’ properties and impacts. NIST works to improve measurements ...
www.nist.gov
December 4, 2025 at 8:17 PM
Our team at NIST's Center for AI Standards and Innovation (CAISI) just released a blog post with open questions for AI measurement science:
www.nist.gov/blogs/caisi-...
www.nist.gov/blogs/caisi-...
Reposted by Ryan Steed
US CAISI (the equivalent of the US "AI Safety Institute") just put out their approach to AI measurement & there's such a significant portion on construct validity (nist.gov/blogs/caisi-...).
Great to see this after ongoing advocating about this issue (arxiv.org/abs/2511.04703)!
Great to see this after ongoing advocating about this issue (arxiv.org/abs/2511.04703)!
December 4, 2025 at 1:50 PM
US CAISI (the equivalent of the US "AI Safety Institute") just put out their approach to AI measurement & there's such a significant portion on construct validity (nist.gov/blogs/caisi-...).
Great to see this after ongoing advocating about this issue (arxiv.org/abs/2511.04703)!
Great to see this after ongoing advocating about this issue (arxiv.org/abs/2511.04703)!
Reposted by Ryan Steed
After having such a great time at #CHI2025 and #FAccT2025, I wanted to share some of my favorite recent papers here!
I'll aim to post new ones throughout the summer and will tag all the authors I can find on Bsky. Please feel welcome to chime in with thoughts / paper recs / etc.!!
🧵⬇️:
I'll aim to post new ones throughout the summer and will tag all the authors I can find on Bsky. Please feel welcome to chime in with thoughts / paper recs / etc.!!
🧵⬇️:
July 14, 2025 at 5:03 PM
After having such a great time at #CHI2025 and #FAccT2025, I wanted to share some of my favorite recent papers here!
I'll aim to post new ones throughout the summer and will tag all the authors I can find on Bsky. Please feel welcome to chime in with thoughts / paper recs / etc.!!
🧵⬇️:
I'll aim to post new ones throughout the summer and will tag all the authors I can find on Bsky. Please feel welcome to chime in with thoughts / paper recs / etc.!!
🧵⬇️:
Reposted by Ryan Steed
Q: What do school buses, desktop computers, and AI have in common?
A: The same decades-old laws and processes apply when governments go to purchase them.
Our new #FAccT2025 paper examines how these legacy public procurement norms apply to AI. 🧵
A: The same decades-old laws and processes apply when governments go to purchase them.
Our new #FAccT2025 paper examines how these legacy public procurement norms apply to AI. 🧵
May 14, 2025 at 2:34 AM
Q: What do school buses, desktop computers, and AI have in common?
A: The same decades-old laws and processes apply when governments go to purchase them.
Our new #FAccT2025 paper examines how these legacy public procurement norms apply to AI. 🧵
A: The same decades-old laws and processes apply when governments go to purchase them.
Our new #FAccT2025 paper examines how these legacy public procurement norms apply to AI. 🧵
Reposted by Ryan Steed
Excited to present "Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling" at #CHI2025 tomorrow(today)!
🗓 Tue, 29 Apr | 9:48–10:00 AM JST (Mon, 28 Apr | 8:48–9:00 PM ET)
📍 G401 (Pacifico North 4F)
📄 dl.acm.org/doi/10.1145/...
🗓 Tue, 29 Apr | 9:48–10:00 AM JST (Mon, 28 Apr | 8:48–9:00 PM ET)
📍 G401 (Pacifico North 4F)
📄 dl.acm.org/doi/10.1145/...
April 28, 2025 at 11:26 AM
Excited to present "Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling" at #CHI2025 tomorrow(today)!
🗓 Tue, 29 Apr | 9:48–10:00 AM JST (Mon, 28 Apr | 8:48–9:00 PM ET)
📍 G401 (Pacifico North 4F)
📄 dl.acm.org/doi/10.1145/...
🗓 Tue, 29 Apr | 9:48–10:00 AM JST (Mon, 28 Apr | 8:48–9:00 PM ET)
📍 G401 (Pacifico North 4F)
📄 dl.acm.org/doi/10.1145/...
Reposted by Ryan Steed
Our lecture topic in our ML eval class today. ✨ Evals have always played an important socio technical role in policy decision-making and surprisingly little changes when you add "AI" to the mix
Rossi’s Metallic Rules and why it’s hard to measure benefit when evaluating social programs.
Rossi's Metallic Rules
Why do evaluations tend to find that social programs don't work?
www.argmin.net
April 22, 2025 at 3:24 PM
Our lecture topic in our ML eval class today. ✨ Evals have always played an important socio technical role in policy decision-making and surprisingly little changes when you add "AI" to the mix
Reposted by Ryan Steed
Great webinar featuring Alessandro Acquisti on "The Economics of Privacy at a Crossroads." Alessandro has such a wide view of the privacy literature: both its history and inside & outside of economics.
www.youtube.com/watch?v=ecc9...
www.youtube.com/watch?v=ecc9...
The Economics of Privacy at a Crossroads | Luohan Webinar
YouTube video by Luohan Academy
www.youtube.com
March 19, 2025 at 5:12 PM
Great webinar featuring Alessandro Acquisti on "The Economics of Privacy at a Crossroads." Alessandro has such a wide view of the privacy literature: both its history and inside & outside of economics.
www.youtube.com/watch?v=ecc9...
www.youtube.com/watch?v=ecc9...
Reposted by Ryan Steed
I was in Paris last week for the #AIActionSummit & was honoured to participate in the “AI in public interest” panel.
👇🏾 my thoughts & reflections on what AI in public interest is/isn’t & some concrete steps/initiatives for `bending the arc of AI towards the public interest' aial.ie/pages/aiparis/
👇🏾 my thoughts & reflections on what AI in public interest is/isn’t & some concrete steps/initiatives for `bending the arc of AI towards the public interest' aial.ie/pages/aiparis/
Bending the arc of AI towards the public interest
By Abeba Birhane, 18/02/2025
Following the first in Bletchley Park in 2023 and the second in Seoul in 2024, the third AI Action Summit took place in February 2025 in Paris. In the context of previous ...
aial.ie
February 18, 2025 at 1:17 PM
I was in Paris last week for the #AIActionSummit & was honoured to participate in the “AI in public interest” panel.
👇🏾 my thoughts & reflections on what AI in public interest is/isn’t & some concrete steps/initiatives for `bending the arc of AI towards the public interest' aial.ie/pages/aiparis/
👇🏾 my thoughts & reflections on what AI in public interest is/isn’t & some concrete steps/initiatives for `bending the arc of AI towards the public interest' aial.ie/pages/aiparis/
Reposted by Ryan Steed
When Toledo police rolled out their Fusus system, allowing officers to tap into the live feeds of privately owned cameras, they promised to only use the power in emergency situations.
We obtained data that tells a very different story about when, and who, TPD watches.
gizmodo.com/clearly-disc...
We obtained data that tells a very different story about when, and who, TPD watches.
gizmodo.com/clearly-disc...
‘Clearly Discrimination’: How a City Uses Fusus to Spy on Its Poorest Residents
Fusus’s technology allows police to tap into live feeds from public and privately owned surveillance cameras. In Toledo, Ohio, cops use the power to watch one particular type of location.
gizmodo.com
February 11, 2025 at 2:27 PM
When Toledo police rolled out their Fusus system, allowing officers to tap into the live feeds of privately owned cameras, they promised to only use the power in emergency situations.
We obtained data that tells a very different story about when, and who, TPD watches.
gizmodo.com/clearly-disc...
We obtained data that tells a very different story about when, and who, TPD watches.
gizmodo.com/clearly-disc...
Reposted by Ryan Steed
As someone who has reported on AI for 7 years and covered China tech as well, I think the biggest lesson to be drawn from DeepSeek is the huge cracks it illustrates with the current dominant paradigm of AI development. A long thread. 1/
January 27, 2025 at 2:12 PM
As someone who has reported on AI for 7 years and covered China tech as well, I think the biggest lesson to be drawn from DeepSeek is the huge cracks it illustrates with the current dominant paradigm of AI development. A long thread. 1/
Reposted by Ryan Steed
How often does the FTC base its enforcement proceedings on third party research? Often! ftcreverse.engineering
FTC RE
ftcreverse.engineering
January 11, 2025 at 8:30 PM
How often does the FTC base its enforcement proceedings on third party research? Often! ftcreverse.engineering
Reposted by Ryan Steed
oh, this is out and they make me sound like an angel but a really good read
“We are not evaluating systems for some hypothetical, potential risks in the future,” Birhane says
www.fastcompany.com/91238006/how...
“We are not evaluating systems for some hypothetical, potential risks in the future,” Birhane says
www.fastcompany.com/91238006/how...
How Abeba Birhane is cleaning up AI’s dirty data
“We are not evaluating systems for some hypothetical, potential risks in the future. These audits are uncovering actual real issues, real problems, whether it's racism, sexism, or encoding of stereoty...
www.fastcompany.com
December 10, 2024 at 7:01 PM
oh, this is out and they make me sound like an angel but a really good read
“We are not evaluating systems for some hypothetical, potential risks in the future,” Birhane says
www.fastcompany.com/91238006/how...
“We are not evaluating systems for some hypothetical, potential risks in the future,” Birhane says
www.fastcompany.com/91238006/how...
Reposted by Ryan Steed
Come work with @jessicahullman.bsky.social and I as a postdoc! We'll apply approaches from HCI, AI/ML, statistics, and decision theory to design new tools & methods for evaluating human-AI decision-making (e.g., tools to elicit, represent, & validate specifications of real-world decision problems).
I'm seeking a postdoc to work with me and @kenholstein.bsky.social on evaluating AI/ML decision support for human experts:
statmodeling.stat.columbia.edu/2024/12/10/p...
P.S. I'll be at NeurIPS Thurs-Mon. Happy to talk about this position or related mutual interests!
Please repost 🙏
statmodeling.stat.columbia.edu/2024/12/10/p...
P.S. I'll be at NeurIPS Thurs-Mon. Happy to talk about this position or related mutual interests!
Please repost 🙏
Postdoc position at Northwestern on evaluating AI/ML decision support | Statistical Modeling, Causal Inference, and Social Science
statmodeling.stat.columbia.edu
December 10, 2024 at 8:10 PM
Come work with @jessicahullman.bsky.social and I as a postdoc! We'll apply approaches from HCI, AI/ML, statistics, and decision theory to design new tools & methods for evaluating human-AI decision-making (e.g., tools to elicit, represent, & validate specifications of real-world decision problems).
Reposted by Ryan Steed
tough morning for data brokers, great morning for people
in addition to the CFPB FCRA rule (bsky.app/profile/benw...), the FTC announced a case against one who sold location data of visits to healthcare and places of worship, securing a ban on sale of this type of data
www.ftc.gov/news-events/...
in addition to the CFPB FCRA rule (bsky.app/profile/benw...), the FTC announced a case against one who sold location data of visits to healthcare and places of worship, securing a ban on sale of this type of data
www.ftc.gov/news-events/...
December 3, 2024 at 2:32 PM
tough morning for data brokers, great morning for people
in addition to the CFPB FCRA rule (bsky.app/profile/benw...), the FTC announced a case against one who sold location data of visits to healthcare and places of worship, securing a ban on sale of this type of data
www.ftc.gov/news-events/...
in addition to the CFPB FCRA rule (bsky.app/profile/benw...), the FTC announced a case against one who sold location data of visits to healthcare and places of worship, securing a ban on sale of this type of data
www.ftc.gov/news-events/...
Reposted by Ryan Steed
The proposed consent order with IntelliVision Technologies would prevent the company “from making misleading claims about its software. That would include any misleading statements about its performance identifying people of different genders, ethnicities, and skin tones.”
The FTC is cracking down on a firm that claims its AI-powered face recognition tech has 'zero' bias
A software provider won’t be allowed to lie about the accuracy of its artificial intelligence-powered facial recognition technology under a new proposed order from the Biden administration.
www.yahoo.com
December 3, 2024 at 6:58 PM
The proposed consent order with IntelliVision Technologies would prevent the company “from making misleading claims about its software. That would include any misleading statements about its performance identifying people of different genders, ethnicities, and skin tones.”
Something (someone) to be thankful for today — very excited to see this!!
Friends, this week marks a monumental step forward in my path towards pushing for a greater ecology of accountability in the age of AI. Thrilled to officially announce the AI Accountability Lab (AIAL) @theaial.bsky.social at Trinity College Dublin is launching today, Thursday Nov 28, 2024 aial.ie
AI Accountability Lab
Stewarding a greater ecology of accountability in the age of AI
aial.ie
November 28, 2024 at 2:18 PM
Something (someone) to be thankful for today — very excited to see this!!
Reposted by Ryan Steed
Friends, this week marks a monumental step forward in my path towards pushing for a greater ecology of accountability in the age of AI. Thrilled to officially announce the AI Accountability Lab (AIAL) @theaial.bsky.social at Trinity College Dublin is launching today, Thursday Nov 28, 2024 aial.ie
AI Accountability Lab
Stewarding a greater ecology of accountability in the age of AI
aial.ie
November 28, 2024 at 10:16 AM
Friends, this week marks a monumental step forward in my path towards pushing for a greater ecology of accountability in the age of AI. Thrilled to officially announce the AI Accountability Lab (AIAL) @theaial.bsky.social at Trinity College Dublin is launching today, Thursday Nov 28, 2024 aial.ie
Reposted by Ryan Steed
Professors and researchers: are you booking speakers for the Spring semester? I have talks and workshops on data-driven investigations using crowdsourcing with WhatsApp, auditing black box AI systems and advanced web scraping.
inspectelement.org/apis.html
gist.github.com/yinleon/4e39...
inspectelement.org/apis.html
gist.github.com/yinleon/4e39...
Finding Undocumented APIs
Introduction, case studies, and exercises for finding and using undocumented APIs hidden in plain sight.
inspectelement.org
November 20, 2024 at 8:42 PM
Professors and researchers: are you booking speakers for the Spring semester? I have talks and workshops on data-driven investigations using crowdsourcing with WhatsApp, auditing black box AI systems and advanced web scraping.
inspectelement.org/apis.html
gist.github.com/yinleon/4e39...
inspectelement.org/apis.html
gist.github.com/yinleon/4e39...
Reposted by Ryan Steed
Differential privacy is a SOTA approach for protecting individual privacy in data releases. But how it interacts w/common expectations & workflows can be hard to judge.
New article for policy-makers w/P. Nanayakkara summarizes some gaps between theory & practice
journals.sagepub.com/doi/abs/10.1...
New article for policy-makers w/P. Nanayakkara summarizes some gaps between theory & practice
journals.sagepub.com/doi/abs/10.1...
Sage Journals: Discover world-class research
Subscription and open access journals from Sage, the world's leading independent academic publisher.
journals.sagepub.com
November 19, 2024 at 6:09 PM
Differential privacy is a SOTA approach for protecting individual privacy in data releases. But how it interacts w/common expectations & workflows can be hard to judge.
New article for policy-makers w/P. Nanayakkara summarizes some gaps between theory & practice
journals.sagepub.com/doi/abs/10.1...
New article for policy-makers w/P. Nanayakkara summarizes some gaps between theory & practice
journals.sagepub.com/doi/abs/10.1...