@meetbryce.bsky.social
"We need to build evals" is the new "we need to write tests"
Everyone agrees. Almost nobody does it. It feels too heavy & if you try to do everything all at once, it is!
Here's what actually works:
Before shipping:
→ 15-20 test scenarios
→ Read every trace
→ Note what breaks
Everyone agrees. Almost nobody does it. It feels too heavy & if you try to do everything all at once, it is!
Here's what actually works:
Before shipping:
→ 15-20 test scenarios
→ Read every trace
→ Note what breaks
January 20, 2026 at 1:37 AM
"We need to build evals" is the new "we need to write tests"
Everyone agrees. Almost nobody does it. It feels too heavy & if you try to do everything all at once, it is!
Here's what actually works:
Before shipping:
→ 15-20 test scenarios
→ Read every trace
→ Note what breaks
Everyone agrees. Almost nobody does it. It feels too heavy & if you try to do everything all at once, it is!
Here's what actually works:
Before shipping:
→ 15-20 test scenarios
→ Read every trace
→ Note what breaks
i have a lot of content i want to consume but never makes it to the top of my to read list. anyone have a good solution/prompt to distilling it down and extracting the good stuff? best i’ve got at the moment is pushing it to NotebookLM 🎧 (which at this point i may just automate)
December 28, 2025 at 9:11 PM
i have a lot of content i want to consume but never makes it to the top of my to read list. anyone have a good solution/prompt to distilling it down and extracting the good stuff? best i’ve got at the moment is pushing it to NotebookLM 🎧 (which at this point i may just automate)
1. figure out how to make great AI products
2. figure out which problems & solutions are worth doing everything required to make it a great product.
not every feature is worthy of the full evals treatment and making that judgment call is a key PM responsibility.
2. figure out which problems & solutions are worth doing everything required to make it a great product.
not every feature is worthy of the full evals treatment and making that judgment call is a key PM responsibility.
September 10, 2025 at 3:50 PM
1. figure out how to make great AI products
2. figure out which problems & solutions are worth doing everything required to make it a great product.
not every feature is worthy of the full evals treatment and making that judgment call is a key PM responsibility.
2. figure out which problems & solutions are worth doing everything required to make it a great product.
not every feature is worthy of the full evals treatment and making that judgment call is a key PM responsibility.
Free startup idea if you're a fast mover – create a US-hosted (provably) iOS app for Deepseek R1. Their own app is free, but with all the concerns about CCP-interests, there's surely a market with the willingness to pay. If I had the time & energy I'd totally being doing this myself tonight.
January 27, 2025 at 6:00 AM
Free startup idea if you're a fast mover – create a US-hosted (provably) iOS app for Deepseek R1. Their own app is free, but with all the concerns about CCP-interests, there's surely a market with the willingness to pay. If I had the time & energy I'd totally being doing this myself tonight.