Lorin Hochstein
@norootcause.surfingcomplexity.com
1.7K followers 830 following 1.3K posts
Student of complex systems failures, resilience engineering, cognitive systems engineering. Will talk your ear off about @resilienceinsoftware.org
Posts Media Videos Starter Packs
Reposted by Lorin Hochstein
snowytrees.dev
Google blacklisted the iCloud private relay ip addresses. Been broken for ages while now. On iOS safari you need to press the “show ip address” option and then try searching again and it works.
norootcause.surfingcomplexity.com
It was the private relay. When I disable it (show IP address), Google Scholar works. When I re-enable it (hide IP address), I immediately get the error message again.
norootcause.surfingcomplexity.com
This was it! It works!!! Thank you!
norootcause.surfingcomplexity.com
Nope, Google search works fine. I only get it on Google Scholar. No idea how my phone got flagged as a bot.
norootcause.surfingcomplexity.com
I can't access Google Scholar over my phone (it gives me an error when I try to search or click a link on a paper), and I have no idea why.
norootcause.surfingcomplexity.com
See also: "The new feature was (intentionally) gradually enabled via configuration rather than a distinct deployment. While this is intended as safety, it also made it harder to immediately connect the outage to the change". www.pagerduty.com/eng/august-2...
August 28 Kafka Outages – What Happened and How We’re Improving
At 3:53 UTC on August 28, 2025 a failure on one of PagerDuty’s message queuing systems (Kafka) triggered cascading issues that disrupted or delayed the...
www.pagerduty.com
norootcause.surfingcomplexity.com
More generally: beware of approaches that make easy incidents easier but hard incidents harder. The trade-off might still be worth it, but acknowledge the downside risk!
norootcause.surfingcomplexity.com
Similar. I didn’t associate the failure mode with flow rate at all. Instead, it was a dep bump that looked low risk that to a surprising behavior change.
norootcause.surfingcomplexity.com
Good public incident write-up from UptimeLabs. Quick notes:

* A patch-level dependency update unexpectedly changed behavior.
* Failure mode: saturation
* The role of health checks (a tool to improve availability!) in enabling this incident.

uptimelabs.io/when-fast-fl...
When Fast Flow Delivers A Real Blow: A PIR - Uptime Labs
An honest Post Incident Review: how fast flow and continuous delivery caused an outage, and why progress over perfection drives resilience.
uptimelabs.io
Reposted by Lorin Hochstein
uncleduke1969.bsky.social
and he would have gotten away with it too if it weren’t for those meddling kids
Picture of cemetery with headline reading “Man fined for pretending to be ghost in Portsmouth cemetery.”
norootcause.surfingcomplexity.com
I would not have gotten my PhD if this rule was in effect when I was a student. (I started off in an EE PhD program at Boston University, decided I wanted to do CS instead, and left BU with an MSEE for U. Maryland, where I got my PhD. That was six years on my student visa).
crampell.bsky.social
Trump admin planning to change student visas from lasting for duration of academic program to fixed 4-yr term, and then much harder to renew
Could destroy US ability to attract global talent, particularly those seeking advanced degrees in STEM. The median time to complete a PhD is 5.7 yrs per NSF.
Trump Deals A New Immigration Blow To International Students
Trump officials have proposed a new rule limiting international students to fixed periods of entry, making a U.S. education more precarious.
www.forbes.com
norootcause.surfingcomplexity.com
prophlegmatic, on the other hand…
Reposted by Lorin Hochstein
uncleduke1969.bsky.social
pretty sure the fire pigeons aren’t gonna care about your silly little sign
A posted sign with a silhouette of a pigeon with what looks like fire behind it, with a red circle around it and line through it, and the word “No” below.
norootcause.surfingcomplexity.com
New blog post: the problems that accountability can't fix.

surfingcomplexity.blog/2025/08/23/t...
norootcause.surfingcomplexity.com
“When the following Director of Engineering was asked about the relationship between OceanGate’s engineering and operations teams, he described it as severely dysfunctional, stating, ‘They didn't talk. They hated each other.’” – from U.S. Coast Guard the Marine Board Investigation report