Bryan and Adam reflect on Oxide and Friends in 2025--favorite moments, episodes, and images. Happy new year and see you in 2026! Your hosts are Bryan Cantrill and Adam Leventhal. Some of the topics we hit on, in the order that we hit them: * RFD 576: Using LLMs at Oxide (hacker news comments) * OxF: Oxide and Friends 6/2/2025 -- AI Discourse with Steve Klabnik (around 1:08:00) * Shell Game podcast * OxF s05e12 – Hell is other networks — April 4, 2025 * "No Egress" was a ChatGPT joke! * OxF s05e33 – A Grown-up ZFS Data Corruption Bug — November 26, 2025 * Simpsons scene deleted in syndication * OxF s05e29 – AI in Higher Education with Michael Littman — October 17, 2025 * OxF s05e28 – Systems Software in the Large * OxF s05e18 – AI, Materials, and Fraud with Ben Shindel * OxF s04e21 – Adventures in Data Corruption * "Duck season, Fire!" * MLG Airhorn aka "the jj airhorn" * OxF s05e31 – Futurelock * Laura's blog post: A disappearing Service Processor * OxF s05e34 – Death by Uptime * "Painfully concrete" - ChatGPT * OxF s05e03 – Holistic Engineering with Robert Mustacchi * OxF s05e30 – RIP USENIX ATC * Team DTrace meets Dennis Richie, redux * "Fart Boy" * OxF s05e?? – Books in the Box V — The latest annual book recommendation episode. * Oxide Bingo by John Holloway * OxF s05e16 – Scaling Manufacturing * OxF s05e22 – Founder vs. Investor * OxF s05e27 – Character Limit * Stretch goal for 2026: finally a C&D * OxF s05e24 – Diving In * Bryan's blog: College Baseball, Venture Capital, and the Long Maybe * OxF s03e31 – Hiring Processes with Gergely Orosz * OxF s04e06 – Crucible: The Oxide Storage Service * "Don't worry, Alan, no one will listen" -> one of our most popular episodes * OxF s04e23 – RFDs: The Backbone of Oxide * Office Space: Michael Bolton as AI em-dash * "This isn't nostalgia, it's epistemology" - ChatGPT * "Weaponized weariness" - ChatGPT * "Dry Fatalism" - ChatGPT * "Cougar turned in his wings..." * OxF s05e16 – Solutions Software Engineering with Matthew Sanabria * Alexander Hamilton: amazing. Also the world's pre-eminent subtweeter and blogger? If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

We hit a new (and disturbing!) failure mode recently when a production rack that had been up for several months saw every (!) compute sled's service processor become simultaneously unresponsive. Bryan and Adam were joined by the members of the Oxide team who debugged the vexing issue -- and reached its surprising root cause. In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Cliff Biffle, Matt Keeter, and Will Chandler. Previously, on Oxide and Friends: * OxF s05e03 – Holistic Engineering with Robert Mustacchi * OxF s04e14 – Rebooting a datacenter: A decade later * OxF s01e26 – The Pragmatism of Hubris * OxF s05e20 – Debugger-Driven Development (omdb) * OxF s05e07 – Transparency in Hardware/Software Interfaces * OxF s05e31 – Futurelock * OxF s05e33 – A Grown-up ZFS Data Corruption Bug Some of the topics we hit on, in the order that we hit them: * hubris #2304: STM32H7 Ethernet driver stops yielding CPU after many packets * gist — Summarizing the Hubris side of investigations * Matt's blog: Hunting a spooky ethernet driver bug If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Hey hey! We recently tripped over a ZFS data corruption bug–introduced over 18 years ago! Bryan and Adam discuss with members of the Oxide team as well as Matt Ahrens, the co-inventor of ZFS. In addition to Bryan Cantrill and Adam Leventhal, speakers included Alan Hanson, Matt Keeter, Andy Fiddaman, James MacMahon, and special guest, Matt Ahrens. Previously, on Oxide and Friends: * OxF s4e6 - Crucible: the Oxide Storage Service * OxF s5e28 - Systems Software in the Large Some of the topics we hit on, in the order that we hit them: * ZFS fsync can trigger ZIL transaction reordering and data corruption * RFD 177: Implementation of Data Storage * the "fix" that introduced data corruption * PRs needed! If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Oxide founders, Bryan and Steve, as well as Oxide investor, Seth Winterroth, were joined by Liz Zalman and Jerry Neumann, authors of the book Founder vs. Investor, discussing the collaboration and conflict in company formation. Adam was also present. In addition to Bryan Cantrill and Adam Leventhal, our guests included Liz Zalman, Jerry Neumann, Seth Winterroth (Oxide investor), and Steve Tuck (Oxide founder / CEO). Some of the topics we hit on, in the order that we hit them: * Founder vs. Investor * Topic * [@M:SS](link into recording) _Leventhal's Conundrum_ * PRs needed! If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

We're big users of async Rust at Oxide, and recently we found (another) very odd and hard to debug pathology related to async Rust that we dubbed "Futurelock". Oxide engineers who diagnosed the problem join Bryan and Adam to describe Futurelock and discuss methods to identify and avoid it. In addition to Bryan Cantrill and Adam Leventhal, speakers included our Oxide colleagues Dave Pacheco, John Gallagher, Rain Paharia. Sean Klein, and Eliza Weisman. Previously, on Oxide and Friends: * OxF s05e22 - When Async Attacks! * OxF s05e26 - Technological Revolutions with Jerry Neumann Some of the topics we hit on, in the order that we hit them: * Oxide RFD 609: Futurelock * PRs needed! If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Revisiting an annual tradition--Books in the Box! Bryan, Adam, and the Oxide Friends share book recommendations (and--sometimes--anti-recommendations). Take a listen if you're looking for your next read. In addition to Bryan Cantrill and Adam Leventhal, we were joined by some guests noted below: Previously, on Oxide and Friends: * OxF s05e06 - A Half‑Century of Silicon Valley with Randy Shoup * OxF s05e26 - Technological Revolutions with Jerry Neumann * OxF s04e03 - Fork in the Road for Terraform * OxF s01e16 - The Books in the Box * OxF s02e18 - Books in the Box Redux * OxF s03e22 - Books in the Box III * OxF s04e28 - Books in the Box IV Other Notes: * Princeton Review: Happiest Students * UMass Dining Named Best Campus Food by The Princeton Review * CHM Oral Histories * Night Rider (and K.I.T.T.) From Bryan and Adam (and others) * The Mouse Driver Chronicles * Fumbling the Future * Slingshot * Chip War * Technological Revolutions and Financial Capital @bcantrill (_economics book recommendation_) * Snow Crash (_another Neal Stephenson book_) * The Big Short * Reinventing The Wheel * Eccentric Orbits (_recommended by listener_) * Language Machines Cultural AI and the End of Remainder Humanism (_recommended by listener_) * Molly White's **review ** of Read Write Own * Careless People * **NOT A RECOMMENDATION** If Anyone Builds It, Everyone Dies (**If you are Molly White, please destroy this for us!**) * Surreal Numbers by Knuth (_recommended by listeners_) From Oliver Herman * Open Circuits * Systems Performance * Why We're Getting Poorer * Termination Shock From Tom Lyon * From Airline Reservations to Sonic the Hedgehog * See also Systems We Love: Life of an Airline Flight * The War of Don Emmanuel's Nether Parts * The NVIDIA Way From Dan McDonald * Inventing the Renaissance * Charles Sumner: Conscience of a Nation If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

LLMs have had a dramatic impact on education. There are obvious reasons for concern, but what about the less obvious opportunities afforded by LLMs? Bryan and Adam were joined by Michael Littman, professor at Brown University and Associate Provost for AI, to talk about his role advising the university on productive, innovative, creative uses for AI in higher education. In addition to Bryan Cantrill and Adam Leventhal, our special guest was Michael Littman. Previous, on Oxide and Friends: * OxF s01e18 - Dijkstra’s Tweetstorm * OxF s04e02 - Open Source LLMs with Simon Willison * OxF s05e18 - AI, Materials, and Fraud with Ben Shindel Some of the topics we hit on, in the order that we hit them: * Michael's home page * Leslie Kaelbling * Computing Up: Rich Sutton * PRs needed! If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Dave Pacheco is leading Oxide's multi-year effort around full-system update. He recently gave a talk about his experience leading that project, the complexities of designing the system and organizing the team. Dave, Bryan, and Adam discuss the project, the many sources of leadership, and the often underestimated peril of "organizational procrastination". In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague, Dave Pacheco. Previously on OxF: * OxF s05e21 - Rebooting a Datacenter: A Decade Later * OxF s01e09 - Agile + 20 * OxF s04e11 - A Baseball Startup with Paul Freedman and Bryan Carmel Some of the topics we hit on, in the order that we hit them: * Dave's talk: Path to self-service update (slides) * Fire trucks dousing the champion Ballers * Bryan's talk: Debugging Under Fire * Roger Faulkner: "I'm not here to make it perfect; I'm here to make it better" * Mid-recording earthquake If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam were joined by members of the Oxide manufacturing team to talk about all that goes into ramping up production, from people and processes to expanding the team and refining inefficiencies. It's a great problem to have! In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, CJ Mendes, Kirstin Neira, Erik Anderson, Aaron Hartwig, and Doug Wibben. Previously on Oxide and Friends... * OxF s03e20 - Tales from Manufacturing: Shipping Rack 1 Some of the topics we hit on, in the order that we hit them: * Topic * [@M:SS](link into recording) _Leventhal's Conundrum_ * PRs needed! If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Jerry Neumann joined Bryan and Adam to discuss his blog post from 2015, examining the work of Carlota Perez on technological revolutions. These waves have similarities, in particular: frenzy, bust, and deployment. Is AI a new wave or the culmination of the IT wave of the last 50 years? In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest, Jerry Neumann. Some of the topics we hit on, in the order that we hit them: * Jerry's 2015 blog post: The Deployment Age * PRs needed! Previous episodes mentioned: * OxF s05e24 - Oxide’s $100M Series B * OxF s05e04 - AI Disruption: DeepSeek and Cerebras If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!