Luca Righetti
@lucarighetti.bsky.social
Research Open_Phil, co-host HearThisIdea. Views my own.
🔸10% Pledge at GivingWhatWeCan.
🔸10% Pledge at GivingWhatWeCan.
Huge thanks to my co-authors:
@Tegan_McCaslin and @jide_alaga (who led this work together), as well as Samira Nedungadi, @seth_donoughe, Tom Reed, @ChrisPainterYup, and @RishiBommasani.
@Tegan_McCaslin and @jide_alaga (who led this work together), as well as Samira Nedungadi, @seth_donoughe, Tom Reed, @ChrisPainterYup, and @RishiBommasani.
September 2, 2025 at 4:03 PM
Huge thanks to my co-authors:
@Tegan_McCaslin and @jide_alaga (who led this work together), as well as Samira Nedungadi, @seth_donoughe, Tom Reed, @ChrisPainterYup, and @RishiBommasani.
@Tegan_McCaslin and @jide_alaga (who led this work together), as well as Samira Nedungadi, @seth_donoughe, Tom Reed, @ChrisPainterYup, and @RishiBommasani.
Still, we believe STREAM can meaningfully raise the bar for model reporting quality.
I'm excited to apply it to recent model cards – stay tuned!
I'm excited to apply it to recent model cards – stay tuned!
September 2, 2025 at 4:03 PM
Still, we believe STREAM can meaningfully raise the bar for model reporting quality.
I'm excited to apply it to recent model cards – stay tuned!
I'm excited to apply it to recent model cards – stay tuned!
Our paper is a "version 1".
The science of evals is evolving, and we want STREAM to evolve with it.
If you have feedback, email us at feedback[at]streamevals[dot]com.
We hope future work expands STREAM beyond ChemBio benchmarks – and we list several ideas in our appendices.
x.com/MariusHobbh...
The science of evals is evolving, and we want STREAM to evolve with it.
If you have feedback, email us at feedback[at]streamevals[dot]com.
We hope future work expands STREAM beyond ChemBio benchmarks – and we list several ideas in our appendices.
x.com/MariusHobbh...
September 2, 2025 at 4:03 PM
Our paper is a "version 1".
The science of evals is evolving, and we want STREAM to evolve with it.
If you have feedback, email us at feedback[at]streamevals[dot]com.
We hope future work expands STREAM beyond ChemBio benchmarks – and we list several ideas in our appendices.
x.com/MariusHobbh...
The science of evals is evolving, and we want STREAM to evolve with it.
If you have feedback, email us at feedback[at]streamevals[dot]com.
We hope future work expands STREAM beyond ChemBio benchmarks – and we list several ideas in our appendices.
x.com/MariusHobbh...
Transparency in AI safety is critical for building trust and advancing our scientific understanding.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
September 2, 2025 at 4:03 PM
Transparency in AI safety is critical for building trust and advancing our scientific understanding.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
Our paper draws on interviews across governments, industry, and academia.
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
September 2, 2025 at 4:03 PM
Our paper draws on interviews across governments, industry, and academia.
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
📄 Read the full paper and find future resources here: streamevals.com/
Here’s a quick overview of what we wrote 🧵
Here’s a quick overview of what we wrote 🧵
September 2, 2025 at 4:03 PM
📄 Read the full paper and find future resources here: streamevals.com/
Here’s a quick overview of what we wrote 🧵
Here’s a quick overview of what we wrote 🧵
I'd love any help to expand this dataset to cover other companies, which I suspect do far worse / don't publish such results at all.
DM me if you'd like to collaborate :))
docs.google.com/spreadsheet...
DM me if you'd like to collaborate :))
docs.google.com/spreadsheet...
AI Safety Cards Publishing Dates
docs.google.com
August 29, 2025 at 5:55 PM
I'd love any help to expand this dataset to cover other companies, which I suspect do far worse / don't publish such results at all.
DM me if you'd like to collaborate :))
docs.google.com/spreadsheet...
DM me if you'd like to collaborate :))
docs.google.com/spreadsheet...
You can read the complete report here --
forecastingresearch.org/ai-enabled-...
Huge thanks to @bridgetw_au and everyone at @Research_FRI for running this survey, as well as to @SecureBio for establishing the "a top team" baseline.
forecastingresearch.org/ai-enabled-...
Huge thanks to @bridgetw_au and everyone at @Research_FRI for running this survey, as well as to @SecureBio for establishing the "a top team" baseline.
Forecasting Biosecurity Risks from LLMs — Forecasting Research Institute
forecastingresearch.org
July 1, 2025 at 3:09 PM
You can read the complete report here --
forecastingresearch.org/ai-enabled-...
Huge thanks to @bridgetw_au and everyone at @Research_FRI for running this survey, as well as to @SecureBio for establishing the "a top team" baseline.
forecastingresearch.org/ai-enabled-...
Huge thanks to @bridgetw_au and everyone at @Research_FRI for running this survey, as well as to @SecureBio for establishing the "a top team" baseline.
Still, there's a clear gap between expert perceptions in biosecurity and actual AI progress.
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
July 1, 2025 at 3:09 PM
Still, there's a clear gap between expert perceptions in biosecurity and actual AI progress.
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
To be clear, "AI matches a top team at VCT" is a high bar. I get why forecasters were surprised.
It means:
• A test designed specifically for bio troubleshooting
• AI outperforming five expert teams (postdocs from elite unis)
• Topics chosen by groups based on their expertise
x.com/DanHendryck...
It means:
• A test designed specifically for bio troubleshooting
• AI outperforming five expert teams (postdocs from elite unis)
• Topics chosen by groups based on their expertise
x.com/DanHendryck...
July 1, 2025 at 3:09 PM
To be clear, "AI matches a top team at VCT" is a high bar. I get why forecasters were surprised.
It means:
• A test designed specifically for bio troubleshooting
• AI outperforming five expert teams (postdocs from elite unis)
• Topics chosen by groups based on their expertise
x.com/DanHendryck...
It means:
• A test designed specifically for bio troubleshooting
• AI outperforming five expert teams (postdocs from elite unis)
• Topics chosen by groups based on their expertise
x.com/DanHendryck...
How much should we trust these results? All forecasts should be treated cautiously. But two things do help:
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
July 1, 2025 at 3:09 PM
How much should we trust these results? All forecasts should be treated cautiously. But two things do help:
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
The good news:
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
July 1, 2025 at 3:09 PM
The good news:
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
I think this is part of a larger trend.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
July 1, 2025 at 3:09 PM
I think this is part of a larger trend.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
Many thanks to my colleague Matthew van der Merwe for doing most of the online sleuthing here (and not on X).
Main sources:
[*] Court documents –static.foxnews.com/foxnews.com...
[*] Youtube –web.archive.org/web/2024090...
[*] Reddit – ihsoyct.github.io/index.html?...
Main sources:
[*] Court documents –static.foxnews.com/foxnews.com...
[*] Youtube –web.archive.org/web/2024090...
[*] Reddit – ihsoyct.github.io/index.html?...
Indict Evolution
web.archive.org
June 9, 2025 at 9:32 AM
Many thanks to my colleague Matthew van der Merwe for doing most of the online sleuthing here (and not on X).
Main sources:
[*] Court documents –static.foxnews.com/foxnews.com...
[*] Youtube –web.archive.org/web/2024090...
[*] Reddit – ihsoyct.github.io/index.html?...
Main sources:
[*] Court documents –static.foxnews.com/foxnews.com...
[*] Youtube –web.archive.org/web/2024090...
[*] Reddit – ihsoyct.github.io/index.html?...
It's worth remembering, US bombings are lower than they used to be. I doubt AI has affected this trend – and it's too early to tell what will happen.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
June 9, 2025 at 9:32 AM
It's worth remembering, US bombings are lower than they used to be. I doubt AI has affected this trend – and it's too early to tell what will happen.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
And you can imagine scenarios far worse.
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
June 9, 2025 at 9:32 AM
And you can imagine scenarios far worse.
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
Notably, a counter-terror strategy is to have police spot suspicious activity in online forums, using that to start investigations and undercover stings.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
June 9, 2025 at 9:32 AM
Notably, a counter-terror strategy is to have police spot suspicious activity in online forums, using that to start investigations and undercover stings.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
By contrast, the suspect's (likely-but-unconfirmed) reddit account also tried asking questions but didn't get any helpful replies.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
June 9, 2025 at 9:32 AM
By contrast, the suspect's (likely-but-unconfirmed) reddit account also tried asking questions but didn't get any helpful replies.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
Still, AI *did* answer many questions about explosives.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
June 9, 2025 at 9:32 AM
Still, AI *did* answer many questions about explosives.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
It’s unclear how counterfactual the AI was.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.
June 9, 2025 at 9:32 AM
It’s unclear how counterfactual the AI was.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.