Definitely not a conspiracy with @beerops & @alicegoldfuss. #monitorama
November 18, 2024 at 11:12 AM
Definitely not a conspiracy with @beerops & @alicegoldfuss. #monitorama
At the #monitorama @grafana workshop, @torkelo points out that evolution is how nature programs.
November 22, 2024 at 12:33 AM
At the #monitorama @grafana workshop, @torkelo points out that evolution is how nature programs.
Well I did it, I made a post on LinkedIn. And since you all deserve to know too: I’ll be hanging out at Monitorama in Portland next week. Tell all of your friends to come talk to me about the future of software observability! (lol… but also seriously)
me: maybe I need to stop ignoring linkedin since it can be an important way to recruit research participants in my field
linkedin: *a string of horrors and indignities*
linkedin: *a string of horrors and indignities*
June 3, 2024 at 8:13 PM
Well I did it, I made a post on LinkedIn. And since you all deserve to know too: I’ll be hanging out at Monitorama in Portland next week. Tell all of your friends to come talk to me about the future of software observability! (lol… but also seriously)
So how can we trace an increase in failures to something downstream in the stack (e.g. ooming database)? [ed: so far, what's being demonstrated is timing correlation rather than _trace_ causation/dependency analysis]
so ideally we surface all the contributing factors #monitorama
so ideally we surface all the contributing factors #monitorama
January 13, 2025 at 8:37 PM
So how can we trace an increase in failures to something downstream in the stack (e.g. ooming database)? [ed: so far, what's being demonstrated is timing correlation rather than _trace_ causation/dependency analysis]
so ideally we surface all the contributing factors #monitorama
so ideally we surface all the contributing factors #monitorama
You also need deep rather than shallow incident data; meaningful data helps us design good experiments (e.g. near-misses, failures of communication/lack of collaboration, surprising failures, lack of observability) #Monitorama
January 13, 2025 at 8:31 PM
You also need deep rather than shallow incident data; meaningful data helps us design good experiments (e.g. near-misses, failures of communication/lack of collaboration, surprising failures, lack of observability) #Monitorama
Why are our graphs more wide than tall, and have time as x-axis? it's easier for us.
Positioning provides meaning -- don't dynamically reflow charts because we form patterns about "third down on the right".
Alerts are charts that we care about. #monitorama
Positioning provides meaning -- don't dynamically reflow charts because we form patterns about "third down on the right".
Alerts are charts that we care about. #monitorama
January 13, 2025 at 8:30 PM
Why are our graphs more wide than tall, and have time as x-axis? it's easier for us.
Positioning provides meaning -- don't dynamically reflow charts because we form patterns about "third down on the right".
Alerts are charts that we care about. #monitorama
Positioning provides meaning -- don't dynamically reflow charts because we form patterns about "third down on the right".
Alerts are charts that we care about. #monitorama
"There's this myth that we need 3 different products. What ends up happening is that you wind up with these different, isolated environments that you funnel developers between..."
Stop gluing things together. Vendors doing this too by creating disparate product :( #monitorama
Stop gluing things together. Vendors doing this too by creating disparate product :( #monitorama
January 13, 2025 at 6:51 PM
"There's this myth that we need 3 different products. What ends up happening is that you wind up with these different, isolated environments that you funnel developers between..."
Stop gluing things together. Vendors doing this too by creating disparate product :( #monitorama
Stop gluing things together. Vendors doing this too by creating disparate product :( #monitorama
When people disagree, that's rich data about exposing how they think and what their models/assumptions are. Don't build a "single coherent" analysis -- that means you're probably wrong, because humans always disagree. #monitorama
January 13, 2025 at 6:45 PM
When people disagree, that's rich data about exposing how they think and what their models/assumptions are. Don't build a "single coherent" analysis -- that means you're probably wrong, because humans always disagree. #monitorama
Have markers on your timeline, as well as next actions. Design deliberately. [fin, a bit over time, so rushing through] #monitorama
January 13, 2025 at 8:30 PM
Have markers on your timeline, as well as next actions. Design deliberately. [fin, a bit over time, so rushing through] #monitorama
Know your data: The stats behind your alerts, Dave McAllister
70% of the time when I mention geometric or harmonic mean, people say “there’s more than one mean?” 100% of the time
#monitorama
70% of the time when I mention geometric or harmonic mean, people say “there’s more than one mean?” 100% of the time
#monitorama
June 30, 2023 at 11:25 PM
Know your data: The stats behind your alerts, Dave McAllister
70% of the time when I mention geometric or harmonic mean, people say “there’s more than one mean?” 100% of the time
#monitorama
70% of the time when I mention geometric or harmonic mean, people say “there’s more than one mean?” 100% of the time
#monitorama
also having next actions/mitigations prepared helps you be better prepared rather than flailing not knowing what to do next to bring the incident under control. #monitorama
January 13, 2025 at 8:34 PM
also having next actions/mitigations prepared helps you be better prepared rather than flailing not knowing what to do next to bring the incident under control. #monitorama
If a human takes their hands off the controls... do you believe that your systems will be running the same way a day later? a week later?
If not, then it means you're doing something to affect the system. What are you doing? #monitorama
If not, then it means you're doing something to affect the system. What are you doing? #monitorama
January 13, 2025 at 6:44 PM
If a human takes their hands off the controls... do you believe that your systems will be running the same way a day later? a week later?
If not, then it means you're doing something to affect the system. What are you doing? #monitorama
If not, then it means you're doing something to affect the system. What are you doing? #monitorama
and they have a bunch of leader/follower processes that are buffering the metrics in realtime, redundantly, to avoid losing data... #Monitorama
January 13, 2025 at 6:58 PM
and they have a bunch of leader/follower processes that are buffering the metrics in realtime, redundantly, to avoid losing data... #Monitorama
Finally, experimenting and checking hypotheses systematically is a superpower. We shouldn't have to make guesswork or assumptions, or rely upon intuition. #monitorama
January 13, 2025 at 6:53 PM
Finally, experimenting and checking hypotheses systematically is a superpower. We shouldn't have to make guesswork or assumptions, or rely upon intuition. #monitorama
Live chaos engineering from Pierre @WavefrontHQ! #monitorama
January 13, 2025 at 6:52 PM
Live chaos engineering from Pierre @WavefrontHQ! #monitorama
They're building alerts for when timeseries stay for 5 min outside Z-scores of 3 (beyond 99% of other values) vs distribution for the past week... #Monitorama
January 13, 2025 at 6:58 PM
They're building alerts for when timeseries stay for 5 min outside Z-scores of 3 (beyond 99% of other values) vs distribution for the past week... #Monitorama
In order to compute our retention rates, we need a data structure that supports percentiles, has a time window, is incrementally updateable, and can be computed in a distributed manner... #monitorama
January 13, 2025 at 6:54 PM
In order to compute our retention rates, we need a data structure that supports percentiles, has a time window, is incrementally updateable, and can be computed in a distributed manner... #monitorama
Next up is @helixoide on improvements to their tracing practice @Dropbox! #monitorama
January 13, 2025 at 6:57 PM
Next up is @helixoide on improvements to their tracing practice @Dropbox! #monitorama
Possibly optional "hackday" != "Grafana Workshop with Torkel Ödegaard". Now I need to try to change my flight home from @Monitorama , omg.
November 22, 2024 at 1:19 AM
Possibly optional "hackday" != "Grafana Workshop with Torkel Ödegaard". Now I need to try to change my flight home from @Monitorama , omg.
Terrifying: "a PHP developer with ssh keys for prod who uses the same passwds everywhere." @fun_cuddles #monitorama
November 22, 2024 at 12:48 AM
Terrifying: "a PHP developer with ssh keys for prod who uses the same passwds everywhere." @fun_cuddles #monitorama
"Most ppl monitor their culture by a yearly employee engagement survey & they're like yup, ppl still work here." @thesarahhagan #monitorama
November 18, 2024 at 10:47 PM
"Most ppl monitor their culture by a yearly employee engagement survey & they're like yup, ppl still work here." @thesarahhagan #monitorama
Trolling with alternate talk titles, by @jessfraz. #monitorama
November 18, 2024 at 10:27 PM
Trolling with alternate talk titles, by @jessfraz. #monitorama
"Three-minute-SLA-satisfying humans don't exist." @alicegoldfuss #monitorama
November 18, 2024 at 11:24 AM
"Three-minute-SLA-satisfying humans don't exist." @alicegoldfuss #monitorama
On moving beyond the “ping ops” model: incident response at @simple isn’t simple. @asenchi #monitorama
November 19, 2024 at 2:07 PM
On moving beyond the “ping ops” model: incident response at @simple isn’t simple. @asenchi #monitorama
4 Pillars of Readiness: Operations, Performance, Agility, Security
Check out @ajdomie's community project to build a robust Operational Readiness value set, and contribute your ideas: https://github.com/adomeier/OperationalReadiness #monitorama
Check out @ajdomie's community project to build a robust Operational Readiness value set, and contribute your ideas: https://github.com/adomeier/OperationalReadiness #monitorama
GitHub - adomeier/OperationalReadiness: Community Project...
Community Project to build a robust Operational Readiness...
github.com
November 17, 2024 at 5:33 PM
4 Pillars of Readiness: Operations, Performance, Agility, Security
Check out @ajdomie's community project to build a robust Operational Readiness value set, and contribute your ideas: https://github.com/adomeier/OperationalReadiness #monitorama
Check out @ajdomie's community project to build a robust Operational Readiness value set, and contribute your ideas: https://github.com/adomeier/OperationalReadiness #monitorama