AJ Stuyvenberg
@ajs.bsky.social
AWS Hero
Staff Eng @ Datadog
Streaming at: twitch.tv/aj_stuyvenberg
Videos at: youtube.com/@astuyve
I write about serverless minutia at aaronstuyvenberg.com/
Staff Eng @ Datadog
Streaming at: twitch.tv/aj_stuyvenberg
Videos at: youtube.com/@astuyve
I write about serverless minutia at aaronstuyvenberg.com/
This includes full Datadog tracing/logs/metrics by the way.
There's no compromising on observability, not even cold starts.
There's no compromising on observability, not even cold starts.
September 18, 2025 at 1:03 PM
This includes full Datadog tracing/logs/metrics by the way.
There's no compromising on observability, not even cold starts.
There's no compromising on observability, not even cold starts.
I use AWS a ton but Lambda still astounds me. Throw some code in a function, send 1m requests as fast as you can.
It ate up all available file descriptors on my little t3 box and still ran 18k RPS with a p99 of 0.3479s. Not many services can go from 0 to 18k RPS instantaneously with this p99.
It ate up all available file descriptors on my little t3 box and still ran 18k RPS with a p99 of 0.3479s. Not many services can go from 0 to 18k RPS instantaneously with this p99.
September 18, 2025 at 1:03 PM
I use AWS a ton but Lambda still astounds me. Throw some code in a function, send 1m requests as fast as you can.
It ate up all available file descriptors on my little t3 box and still ran 18k RPS with a p99 of 0.3479s. Not many services can go from 0 to 18k RPS instantaneously with this p99.
It ate up all available file descriptors on my little t3 box and still ran 18k RPS with a p99 of 0.3479s. Not many services can go from 0 to 18k RPS instantaneously with this p99.
The faster your cold starts are, the cheaper these will be!
August 5, 2025 at 5:51 PM
The faster your cold starts are, the cheaper these will be!
Lambda now charges for init time, so it's useful to count sandboxes which are proactively initialized but never receive a request.
Here's what happens after a 10k request burst. Hundreds of sandbox shutdowns, along with 22 sandboxes which were spun up but never received a request.
Here's what happens after a 10k request burst. Hundreds of sandbox shutdowns, along with 22 sandboxes which were spun up but never received a request.
August 5, 2025 at 5:51 PM
Lambda now charges for init time, so it's useful to count sandboxes which are proactively initialized but never receive a request.
Here's what happens after a 10k request burst. Hundreds of sandbox shutdowns, along with 22 sandboxes which were spun up but never received a request.
Here's what happens after a 10k request burst. Hundreds of sandbox shutdowns, along with 22 sandboxes which were spun up but never received a request.
Happy Lambda Init Billing day to those who celebrate. Fix your cold starts!
August 1, 2025 at 3:04 PM
Happy Lambda Init Billing day to those who celebrate. Fix your cold starts!
NEW: Lambda can now send up to 200mb payloads using response streaming! I assume this is mostly directed at LLM inference workloads, where chatbots can stream large amounts of data over the wire as it becomes available.
August 1, 2025 at 1:10 AM
NEW: Lambda can now send up to 200mb payloads using response streaming! I assume this is mostly directed at LLM inference workloads, where chatbots can stream large amounts of data over the wire as it becomes available.
That said, I'm excited to share that @Datadog's Serverless monitoring product now supports LWA!
Thanks to Harold and AWS Labs for collaborating with us on the PRs, and huge thanks to Alex Gallotta for driving this work.
Thanks to Harold and AWS Labs for collaborating with us on the PRs, and huge thanks to Alex Gallotta for driving this work.
July 30, 2025 at 5:43 PM
That said, I'm excited to share that @Datadog's Serverless monitoring product now supports LWA!
Thanks to Harold and AWS Labs for collaborating with us on the PRs, and huge thanks to Alex Gallotta for driving this work.
Thanks to Harold and AWS Labs for collaborating with us on the PRs, and huge thanks to Alex Gallotta for driving this work.
I've long been an advocate for the Lambda Web Adapter project which lets anyone pretty easily ship an app to Lambda without learning about the event model/API.
Honestly AWS should simply support this natively.
Honestly AWS should simply support this natively.
July 30, 2025 at 5:43 PM
I've long been an advocate for the Lambda Web Adapter project which lets anyone pretty easily ship an app to Lambda without learning about the event model/API.
Honestly AWS should simply support this natively.
Honestly AWS should simply support this natively.
I'm a big fan of continuous profiling/measuring your software against real world use cases. This is also how I often learn about in to new system changes in AWS early, heh.
Great episode of Software Huddle w/ @alexbdebrie: www.youtube.com/watch?v=JAw9...
Great episode of Software Huddle w/ @alexbdebrie: www.youtube.com/watch?v=JAw9...
Operational Excellence Is the Moat with Sam Lambert
Today, Sam Lambert from Planetscale is back for a third time. Planetscale just announced Planetscale Postgres, so we had to get Sam back to tell us how and why they decided to add support for Postgres. It's always great to have Sam on -- he brings great stories about real customers and honest insight about the state of the database industry.
In this episode, we talk about the road to Postgres and how operational excellence is the only true advantage in database providers. Sam walks us through the current Planetscale Postgres offering, along with details on Nova, a new sharded Postgres project that Planetscale is working on. Along the way, we get updates on Planetscale Metal, how demand has been for Planetscale Postgres, and future plans for Planetscale.
*Timestamps*
01:16 Start
06:37 The Timeline
15:15 Not Much IP in the Database Market
21:48 PSBouncer
24:17 Zonal affinity
27:38 Query Insights
29:34 How to sign up
32:02 Convex
34:37 Other data stores?
56:18 Acquisitions
www.youtube.com
July 28, 2025 at 2:59 PM
I'm a big fan of continuous profiling/measuring your software against real world use cases. This is also how I often learn about in to new system changes in AWS early, heh.
Great episode of Software Huddle w/ @alexbdebrie: www.youtube.com/watch?v=JAw9...
Great episode of Software Huddle w/ @alexbdebrie: www.youtube.com/watch?v=JAw9...
"We run benchmarks continually across all of our competitors, not just queries - even connections, ensuring we don't add any latency at all." @isamlambert
Performance is such a competitive advantage which easily slips away if you're not constantly paying attention to it.
Performance is such a competitive advantage which easily slips away if you're not constantly paying attention to it.
July 28, 2025 at 2:59 PM
"We run benchmarks continually across all of our competitors, not just queries - even connections, ensuring we don't add any latency at all." @isamlambert
Performance is such a competitive advantage which easily slips away if you're not constantly paying attention to it.
Performance is such a competitive advantage which easily slips away if you're not constantly paying attention to it.
Reposted by AJ Stuyvenberg
Datadog rewrote its AWS Lambda Extension from #Golang to #Rustlang with no prior Rust experience. @ajs.bsky.social will share how they achieved an 80% Lambda cold start improvement along with a 50% memory footprint reduction at our free and virtual #P99CONF. www.p99conf.io?latest_sfdc_...
#ScyllaDB
#ScyllaDB
July 23, 2025 at 2:17 PM
Datadog rewrote its AWS Lambda Extension from #Golang to #Rustlang with no prior Rust experience. @ajs.bsky.social will share how they achieved an 80% Lambda cold start improvement along with a 50% memory footprint reduction at our free and virtual #P99CONF. www.p99conf.io?latest_sfdc_...
#ScyllaDB
#ScyllaDB
In our case the secret is a Datadog API key which isn't required until we actually flush data, so deferring it to that point saves us over 50ms.
July 22, 2025 at 3:31 PM
In our case the secret is a Datadog API key which isn't required until we actually flush data, so deferring it to that point saves us over 50ms.
Here's another 33% cold start reduction, which comes from deferring expensive decryption calls made to AWS Secrets Manager until the secret is actually needed.
Lazy loading is great!
Lazy loading is great!
July 22, 2025 at 3:31 PM
Here's another 33% cold start reduction, which comes from deferring expensive decryption calls made to AWS Secrets Manager until the secret is actually needed.
Lazy loading is great!
Lazy loading is great!
50% lol was not reading the profile carefully
July 18, 2025 at 4:45 PM
50% lol was not reading the profile carefully
By switching to a memory arena, we preallocate a slab of memory and virtually eliminate the linear growth of malloc syscalls, which cuts down kernel mode switches, improving latency.
Thank you profiling (and jemalloc)!
Thank you profiling (and jemalloc)!
July 18, 2025 at 3:07 PM
By switching to a memory arena, we preallocate a slab of memory and virtually eliminate the linear growth of malloc syscalls, which cuts down kernel mode switches, improving latency.
Thank you profiling (and jemalloc)!
Thank you profiling (and jemalloc)!
Here's how to visualize a 100% memory allocation improvement!
A recent stress test revealed that malloc calls bottlenecked when sending > 100k spans through the API and aggregator pipelines in Lambda.
A recent stress test revealed that malloc calls bottlenecked when sending > 100k spans through the API and aggregator pipelines in Lambda.
July 18, 2025 at 3:07 PM
Here's how to visualize a 100% memory allocation improvement!
A recent stress test revealed that malloc calls bottlenecked when sending > 100k spans through the API and aggregator pipelines in Lambda.
A recent stress test revealed that malloc calls bottlenecked when sending > 100k spans through the API and aggregator pipelines in Lambda.
Now writing a job to a log and then using a subscription filter to run them async is deeply fucking cursed though omg
July 17, 2025 at 5:01 PM
Now writing a job to a log and then using a subscription filter to run them async is deeply fucking cursed though omg
I think OP's intentions were pretty pure until they felt they were mistreated by AWS. So many people end up taking to social media in those instances so in my opinion it was mostly fine.
July 17, 2025 at 5:01 PM
I think OP's intentions were pretty pure until they felt they were mistreated by AWS. So many people end up taking to social media in those instances so in my opinion it was mostly fine.
cc @quinnypig.com, as I saw this post in your newsletter
July 17, 2025 at 4:25 PM
cc @quinnypig.com, as I saw this post in your newsletter
