Ivan Babrou
@mastodon.mastodon.ivan.computer.ap.brid.gy
Doing performance @ Cloudflare.
🌉 bridged from ⁂ https://mastodon.ivan.computer/@mastodon, follow @ap.brid.gy to interact
🌉 bridged from ⁂ https://mastodon.ivan.computer/@mastodon, follow @ap.brid.gy to interact
Looking at devmapper ioctl failures shows limited utility of just 16 frames of LBR. You mostly capture cleanup if you attach to high level functions.
Having bpf accounting enabled doesn't help either.
Having bpf accounting enabled doesn't help either.
February 10, 2026 at 11:56 PM
Looking at devmapper ioctl failures shows limited utility of just 16 frames of LBR. You mostly capture cleanup if you attach to high level functions.
Having bpf accounting enabled doesn't help either.
Having bpf accounting enabled doesn't help either.
Sneak peek: LBR based stacks in tracing spans for failed syscalls through ebpf_exporter.
Think "retsnoop, but more structured". It be attached to existing traces too.
Also pictured: SRSO sadness.
#ebpf #kernel #ebpf_exporter
Think "retsnoop, but more structured". It be attached to existing traces too.
Also pictured: SRSO sadness.
#ebpf #kernel #ebpf_exporter
February 6, 2026 at 3:33 AM
Sneak peek: LBR based stacks in tracing spans for failed syscalls through ebpf_exporter.
Think "retsnoop, but more structured". It be attached to existing traces too.
Also pictured: SRSO sadness.
#ebpf #kernel #ebpf_exporter
Think "retsnoop, but more structured". It be attached to existing traces too.
Also pictured: SRSO sadness.
#ebpf #kernel #ebpf_exporter
If you have a lot of ebpf programs and you have any sort of monitoring that looks at /proc/kallsyms regularly, you might want to have my patch applied: https://lore.kernel.org/bpf/[email protected]/T/#u […]
[Original post on mastodon.ivan.computer]
[Original post on mastodon.ivan.computer]
January 30, 2026 at 6:17 PM
If you have a lot of ebpf programs and you have any sort of monitoring that looks at /proc/kallsyms regularly, you might want to have my patch applied: https://lore.kernel.org/bpf/[email protected]/T/#u […]
[Original post on mastodon.ivan.computer]
[Original post on mastodon.ivan.computer]
... and Rust symbols are in fact shorter demangled if you drop the hashes.
I'm seeing 2.65x longer symbols for ClickHouse and ~0.7-0.9x shorter Rust symbols for our production code.
I'm seeing 2.65x longer symbols for ClickHouse and ~0.7-0.9x shorter Rust symbols for our production code.
January 8, 2026 at 4:38 AM
... and Rust symbols are in fact shorter demangled if you drop the hashes.
I'm seeing 2.65x longer symbols for ClickHouse and ~0.7-0.9x shorter Rust symbols for our production code.
I'm seeing 2.65x longer symbols for ClickHouse and ~0.7-0.9x shorter Rust symbols for our production code.
627k for a ptr::drop_in_place 🫣
It is 753k mangled! Yes, it is longer mangled.
Take that C++, which tops at measly 167k.
It is 753k mangled! Yes, it is longer mangled.
Take that C++, which tops at measly 167k.
January 3, 2026 at 4:58 AM
627k for a ptr::drop_in_place 🫣
It is 753k mangled! Yes, it is longer mangled.
Take that C++, which tops at measly 167k.
It is 753k mangled! Yes, it is longer mangled.
Take that C++, which tops at measly 167k.
Apparently even that is not the limit, because llvm-cxxfilt choked on the biggest ones.
Here's a symbol of 64,039 bytes! Well, a part of it, because it does not fit in the screenshot. It's a part of a single stack trace that is 184,838 bytes long […]
[Original post on mastodon.ivan.computer]
Here's a symbol of 64,039 bytes! Well, a part of it, because it does not fit in the screenshot. It's a part of a single stack trace that is 184,838 bytes long […]
[Original post on mastodon.ivan.computer]
January 3, 2026 at 4:31 AM
Apparently even that is not the limit, because llvm-cxxfilt choked on the biggest ones.
Here's a symbol of 64,039 bytes! Well, a part of it, because it does not fit in the screenshot. It's a part of a single stack trace that is 184,838 bytes long […]
[Original post on mastodon.ivan.computer]
Here's a symbol of 64,039 bytes! Well, a part of it, because it does not fit in the screenshot. It's a part of a single stack trace that is 184,838 bytes long […]
[Original post on mastodon.ivan.computer]
Of course this doesn't even come close to this real C++ symbol from ClickHouse. 17900 bytes!
December 26, 2025 at 8:26 PM
Of course this doesn't even come close to this real C++ symbol from ClickHouse. 17900 bytes!
I'm looking at some outlier profiles and apparently Go function names can look like the following these days. That's 440 bytes!
December 23, 2025 at 1:35 AM
I'm looking at some outlier profiles and apparently Go function names can look like the following these days. That's 440 bytes!
don’t forget to vote if you can, but haven’t yet
November 4, 2025 at 3:49 PM
don’t forget to vote if you can, but haven’t yet
I updated my dmesg_hearbeat kernel module example in rust and it's now easier to build than ever: https://github.com/bobrik/dmesg_heartbeat
On the most recent ubuntu it should build out of the box. If you build your kernel yourself, you might need a single line patch.
#linux #kernel #rust
On the most recent ubuntu it should build out of the box. If you build your kernel yourself, you might need a single line patch.
#linux #kernel #rust
October 5, 2025 at 5:15 AM
I updated my dmesg_hearbeat kernel module example in rust and it's now easier to build than ever: https://github.com/bobrik/dmesg_heartbeat
On the most recent ubuntu it should build out of the box. If you build your kernel yourself, you might need a single line patch.
#linux #kernel #rust
On the most recent ubuntu it should build out of the box. If you build your kernel yourself, you might need a single line patch.
#linux #kernel #rust
As always, you can run this easily in Docker.
Here it is showing which timers run in my vm:
docker run --rm -it --privileged --net host -v /sys/kernel/debug:/sys/kernel/debug:ro -v /sys/fs/cgroup:/sys/fs/cgroup:ro ghcr.io/cloudflare/ebpf_exporter:v2.5.0 --config.dir /examples --config.names timers
Here it is showing which timers run in my vm:
docker run --rm -it --privileged --net host -v /sys/kernel/debug:/sys/kernel/debug:ro -v /sys/fs/cgroup:/sys/fs/cgroup:ro ghcr.io/cloudflare/ebpf_exporter:v2.5.0 --config.dir /examples --config.names timers
July 31, 2025 at 9:15 PM
As always, you can run this easily in Docker.
Here it is showing which timers run in my vm:
docker run --rm -it --privileged --net host -v /sys/kernel/debug:/sys/kernel/debug:ro -v /sys/fs/cgroup:/sys/fs/cgroup:ro ghcr.io/cloudflare/ebpf_exporter:v2.5.0 --config.dir /examples --config.names timers
Here it is showing which timers run in my vm:
docker run --rm -it --privileged --net host -v /sys/kernel/debug:/sys/kernel/debug:ro -v /sys/fs/cgroup:/sys/fs/cgroup:ro ghcr.io/cloudflare/ebpf_exporter:v2.5.0 --config.dir /examples --config.names timers
It's been too long since the last ebpf_exporter release, so I cut v2.5.0 today.
https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.5.0
#ebpfef="/hashtag/ebpf_exporter" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link">#ebpf_exporter #ebpf #bpf #linux #kernel
https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.5.0
#ebpfef="/hashtag/ebpf_exporter" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link">#ebpf_exporter #ebpf #bpf #linux #kernel
July 31, 2025 at 9:06 PM
It's been too long since the last ebpf_exporter release, so I cut v2.5.0 today.
https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.5.0
#ebpfef="/hashtag/ebpf_exporter" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link">#ebpf_exporter #ebpf #bpf #linux #kernel
https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.5.0
#ebpfef="/hashtag/ebpf_exporter" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link">#ebpf_exporter #ebpf #bpf #linux #kernel
I ended up trying dnsmasq that suppresses gateway.icloud.com on my home server and pointing Apple TV at it (manually to bypass DoH). One needs to suppress A, AAAA and importantly HTTPS records to make this work.
It reduces traffic 10000x (!), but […]
[Original post on mastodon.ivan.computer]
It reduces traffic 10000x (!), but […]
[Original post on mastodon.ivan.computer]
July 20, 2025 at 7:04 AM
I ended up trying dnsmasq that suppresses gateway.icloud.com on my home server and pointing Apple TV at it (manually to bypass DoH). One needs to suppress A, AAAA and importantly HTTPS records to make this work.
It reduces traffic 10000x (!), but […]
[Original post on mastodon.ivan.computer]
It reduces traffic 10000x (!), but […]
[Original post on mastodon.ivan.computer]
More blood boiling stuff
July 19, 2025 at 7:04 AM
More blood boiling stuff
Debugging this is absolute pain in the ass.
To add a mitm proxy to inspect traffic, you need to add a profile, which requires switching to wifi. In the end the proxy doesn't work with cryptic Apple specific TLS errors (curl works just fine). It may or […]
[Original post on mastodon.ivan.computer]
To add a mitm proxy to inspect traffic, you need to add a profile, which requires switching to wifi. In the end the proxy doesn't work with cryptic Apple specific TLS errors (curl works just fine). It may or […]
[Original post on mastodon.ivan.computer]
July 15, 2025 at 4:44 AM
Debugging this is absolute pain in the ass.
To add a mitm proxy to inspect traffic, you need to add a profile, which requires switching to wifi. In the end the proxy doesn't work with cryptic Apple specific TLS errors (curl works just fine). It may or […]
[Original post on mastodon.ivan.computer]
To add a mitm proxy to inspect traffic, you need to add a profile, which requires switching to wifi. In the end the proxy doesn't work with cryptic Apple specific TLS errors (curl works just fine). It may or […]
[Original post on mastodon.ivan.computer]
My Apple TV has Siri off, but it doesn't stop it from downloading something Siri related in a loop due to a recurring error, burning through 184GiB a month.
Traffic might be cheap for Apple, but xfinity only let's me download 1200GiB before extorting […]
[Original post on mastodon.ivan.computer]
Traffic might be cheap for Apple, but xfinity only let's me download 1200GiB before extorting […]
[Original post on mastodon.ivan.computer]
July 15, 2025 at 4:39 AM
My Apple TV has Siri off, but it doesn't stop it from downloading something Siri related in a loop due to a recurring error, burning through 184GiB a month.
Traffic might be cheap for Apple, but xfinity only let's me download 1200GiB before extorting […]
[Original post on mastodon.ivan.computer]
Traffic might be cheap for Apple, but xfinity only let's me download 1200GiB before extorting […]
[Original post on mastodon.ivan.computer]
Blood boiling stuff
January 31, 2025 at 7:28 AM
Blood boiling stuff
Fun stuff: #linux #perf can't copy userspace stacks over 64k bytes due to u16 field size limit. You need the stack to do unwinding with dwarf.
I have a #rust program that has a single frame twice as big. It's 3 frames combined into one, but still.
The default stack size to copy in perf is just 8k.
I have a #rust program that has a single frame twice as big. It's 3 frames combined into one, but still.
The default stack size to copy in perf is just 8k.
January 15, 2025 at 3:35 AM
If you use #linux #perf on #rust binaries, I highly recommend you check this out:
* https://github.com/torvalds/linux/commit/c3f8644c21df9b7db97eb70e08e2826368aaafa0
For a decently sized binary (just one!) we observed the following resource consumption on v6.6 and Debian Bookworm:
* https://github.com/torvalds/linux/commit/c3f8644c21df9b7db97eb70e08e2826368aaafa0
For a decently sized binary (just one!) we observed the following resource consumption on v6.6 and Debian Bookworm:
January 14, 2025 at 9:47 PM
If you use #linux #perf on #rust binaries, I highly recommend you check this out:
* https://github.com/torvalds/linux/commit/c3f8644c21df9b7db97eb70e08e2826368aaafa0
For a decently sized binary (just one!) we observed the following resource consumption on v6.6 and Debian Bookworm:
* https://github.com/torvalds/linux/commit/c3f8644c21df9b7db97eb70e08e2826368aaafa0
For a decently sized binary (just one!) we observed the following resource consumption on v6.6 and Debian Bookworm:
This somewhat explains the unnatural feeling when going 50mph downhill 🫨
December 23, 2024 at 1:01 AM
This somewhat explains the unnatural feeling when going 50mph downhill 🫨