Moritz Hoffmann
@antiguru.bsky.social
Computer scientist at @materialize.com. Systems, databases, dataflows. Playing the bass guitar.
Highlighting some of my team's recent work: We've changed Materialize to use swap instead of memory-mapped files, with nice performance and efficiency improvements.
We’ve released a major improvement to our memory spilling infrastructure:
Materialize now uses swap to scale SQL workloads beyond RAM.
✅ Faster hydration
✅ Efficient memory utilization
✅ Bigger workloads supported
Full post from antiguru.bsky.social → bit.ly/46EF2iJ
Materialize now uses swap to scale SQL workloads beyond RAM.
✅ Faster hydration
✅ Efficient memory utilization
✅ Bigger workloads supported
Full post from antiguru.bsky.social → bit.ly/46EF2iJ
September 18, 2025 at 2:24 PM
Highlighting some of my team's recent work: We've changed Materialize to use swap instead of memory-mapped files, with nice performance and efficiency improvements.
Based on Timely 0.24, we released Differential Dataflow 0.17. Compatibility changes plus removing the Filter and Freeze wrappers because they weren't used and difficult to maintain. github.com/TimelyDatafl...
September 15, 2025 at 7:52 PM
Based on Timely 0.24, we released Differential Dataflow 0.17. Compatibility changes plus removing the Filter and Freeze wrappers because they weren't used and difficult to maintain. github.com/TimelyDatafl...
We also added a new way to chain multiple flat-map calls into a single Timely operator. Coming from Rust's iterator API, it's sometimes not obvious that Timely's API has a different cost structure (each transform renders an operator). Here, a builder can mitigate some of the cost.
Introduce builder for flatmap operators by frankmcsherry · Pull Request #704 · TimelyDataflow/timely-dataflow
We discussed this a year back, but I couldn't recall why we didn't move it forward. I copy/pasted the code, and dressed up the documentation and examples. Worth taking a peek!
github.com
September 15, 2025 at 7:44 PM
We also added a new way to chain multiple flat-map calls into a single Timely operator. Coming from Rust's iterator API, it's sometimes not obvious that Timely's API has a different cost structure (each transform renders an operator). Here, a builder can mitigate some of the cost.
I think these change are a great example of taking existing APIs and moving from row-first to batch-of-data. Computers generally seem better at handling data by column rather than row, and these changes allow Timely programs to represent data in their own form.
September 15, 2025 at 7:44 PM
I think these change are a great example of taking existing APIs and moving from row-first to batch-of-data. Computers generally seem better at handling data by column rather than row, and these changes allow Timely programs to represent data in their own form.
A Distributor trait takes the concepts of Exchange, but moves away from row-by-row processing, opening up new kinds of containers that aren't row-first. For example, what is a row in a trie-structured container? The new trait allows to partition by whatever the user wants, not just by a "row".
Distributor trait by antiguru · Pull Request #700 · TimelyDataflow/timely-dataflow
Introduce a distributor that knows how to partition containers across multiple pushers. This moves the partition logic from container builders into a bespoke trait and implementation instead of mix...
github.com
September 15, 2025 at 7:44 PM
A Distributor trait takes the concepts of Exchange, but moves away from row-by-row processing, opening up new kinds of containers that aren't row-first. For example, what is a row in a trie-structured container? The new trait allows to partition by whatever the user wants, not just by a "row".
Previously, the Container trait described behavior that is now separate: Accountable reveals a container's record count for progress tracking; nothing else. For higher-level APIs, IterContainer and DrainContainer reveal their contents (non-)destructively, for container-agnostic row-by-row operators.
Rework container abstractions by antiguru · Pull Request #697 · TimelyDataflow/timely-dataflow
This is another attempt at reworking the container abstractions in Timely.
At the moment, we have two types that a lot of the container abstractions hinge on, Container and ContainerBuilder. Both d...
github.com
September 15, 2025 at 7:44 PM
Previously, the Container trait described behavior that is now separate: Accountable reveals a container's record count for progress tracking; nothing else. For higher-level APIs, IterContainer and DrainContainer reveal their contents (non-)destructively, for container-agnostic row-by-row operators.
We just released Timely Dataflow 0.24! Here are some exciting changes from @frankmcsherry.bsky.social and myself.
The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below.
github.com/TimelyDatafl...
The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below.
github.com/TimelyDatafl...
Release timely-v0.24.0 · TimelyDataflow/timely-dataflow
This version of Timely has some exciting new features.
The Distributor trait offers a generalization of the Exchange type. It allows users to define custom distribution
strategies for routing data...
github.com
September 15, 2025 at 7:44 PM
We just released Timely Dataflow 0.24! Here are some exciting changes from @frankmcsherry.bsky.social and myself.
The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below.
github.com/TimelyDatafl...
The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below.
github.com/TimelyDatafl...
github.com/release-plz/... for handling releases. GitHub CI for builds.
GitHub - release-plz/release-plz: Publish Rust crates from CI with a Release PR.
Publish Rust crates from CI with a Release PR. Contribute to release-plz/release-plz development by creating an account on GitHub.
github.com
August 27, 2025 at 5:19 AM
github.com/release-plz/... for handling releases. GitHub CI for builds.
A lot of performance improvements went into this one!
Materialize 25.2 is here! New features include live freshness reports for all your views, 2.5x faster data product deployment times, and native SQL Server support.
See how these updates can help streamline your operations: bit.ly/44i2hg2
See how these updates can help streamline your operations: bit.ly/44i2hg2
June 25, 2025 at 3:45 PM
A lot of performance improvements went into this one!
Reposted by Moritz Hoffmann
The Materialize engineering team uncovered a rare concurrency bug 🪲in Rust’s 🦀 unbounded channels that could lead to double-free memory errors. After thorough debugging and working with the Rust and crossbeam communities, the fix is now part of @rust-lang.org 1.87.0.
🔗 bit.ly/3Fan1Om
🔗 bit.ly/3Fan1Om
May 15, 2025 at 6:15 PM
The Materialize engineering team uncovered a rare concurrency bug 🪲in Rust’s 🦀 unbounded channels that could lead to double-free memory errors. After thorough debugging and working with the Rust and crossbeam communities, the fix is now part of @rust-lang.org 1.87.0.
🔗 bit.ly/3Fan1Om
🔗 bit.ly/3Fan1Om
Reposted by Moritz Hoffmann
Agents generate more data and place more demand on systems than ever before—and standards like MCP will only accelerate this trend. Learn how Delphi is rethinking how they build data-intensive applications—from the db to the UI: bit.ly/42BZdf9
Scaling queries on agent-produced data: How Delphi transformed its data infrastructure
Join our webinar with Delphi to discover they evolved their data infrastructure to handle the rapid increase in AI-driven interactions with Materialize.
bit.ly
April 11, 2025 at 2:24 PM
Agents generate more data and place more demand on systems than ever before—and standards like MCP will only accelerate this trend. Learn how Delphi is rethinking how they build data-intensive applications—from the db to the UI: bit.ly/42BZdf9
Reposted by Moritz Hoffmann
I'm very excited to share that @materialize.com now has a Self-Managed option, as well as a Community Edition license that lets you play around with the exact same bits but for free.
materialize.com/blog/materia...
materialize.com/blog/materia...
Materialize For Everyone: Introducing Self-Managed and our Free Community Edition
Unlock real-time data transformation with Self-Managed Materialize. Now, deploy Materialize in your own cloud for security, compliance, and performance control—powered by the same battle-tested, incre...
materialize.com
March 11, 2025 at 2:47 PM
I'm very excited to share that @materialize.com now has a Self-Managed option, as well as a Community Edition license that lets you play around with the exact same bits but for free.
materialize.com/blog/materia...
materialize.com/blog/materia...
Jan '24: 1,338 kWh. That was a newly-built apartment in NYC. I don't miss this part!
March 11, 2025 at 3:40 PM
Jan '24: 1,338 kWh. That was a newly-built apartment in NYC. I don't miss this part!
Reposted by Moritz Hoffmann
For interested folks, the RustNYC presentation I gave about the `columnar` crate is now available in the repo: github.com/frankmcsherr...
It's not as good without the talk track, but it's got the benchmark results, you can admire the stylistic watercolors (so therapeutic to make; strong recommend).
It's not as good without the talk track, but it's got the benchmark results, you can admire the stylistic watercolors (so therapeutic to make; strong recommend).
github.com
January 22, 2025 at 8:21 PM
For interested folks, the RustNYC presentation I gave about the `columnar` crate is now available in the repo: github.com/frankmcsherr...
It's not as good without the talk track, but it's got the benchmark results, you can admire the stylistic watercolors (so therapeutic to make; strong recommend).
It's not as good without the talk track, but it's got the benchmark results, you can admire the stylistic watercolors (so therapeutic to make; strong recommend).
We just landed the first use of the columnar crate in Materialize! Stay tuned for more.
github.com/MaterializeI...
github.com/MaterializeI...
January 22, 2025 at 9:17 PM
We just landed the first use of the columnar crate in Materialize! Stay tuned for more.
github.com/MaterializeI...
github.com/MaterializeI...
I wrote a bit about efficient maintenance of temporal filters in @materialize.com. By periodically restarting replicas, we can limit resource requirements (RAM and CPU), all without any other user-visible impact.
materialize.com/blog/replica...
materialize.com/blog/replica...
Replica expiration: Limiting temporal filters' resource requirements
Replica expiration is a new feature in Materialize that limits the resource requirements of temporal filters.
materialize.com
January 15, 2025 at 5:13 PM
I wrote a bit about efficient maintenance of temporal filters in @materialize.com. By periodically restarting replicas, we can limit resource requirements (RAM and CPU), all without any other user-visible impact.
materialize.com/blog/replica...
materialize.com/blog/replica...
An ocelot with an odd shape at the Museum für Naturkunde in Berlin.
December 29, 2024 at 3:19 PM
An ocelot with an odd shape at the Museum für Naturkunde in Berlin.
Reposted by Moritz Hoffmann
I was recently coveted a 10 year journal on jetpens, because that would be cool to do, and realized that I already had one! The blog turns 10 this month, and I thought it might be worth taking a tour through each of the years and seeing what I can jot down about each.
github.com/frankmcsherr...
github.com/frankmcsherr...
github.com
December 23, 2024 at 5:58 PM
I was recently coveted a 10 year journal on jetpens, because that would be cool to do, and realized that I already had one! The blog turns 10 this month, and I thought it might be worth taking a tour through each of the years and seeing what I can jot down about each.
github.com/frankmcsherr...
github.com/frankmcsherr...
Origami tree at the @natural-history.bsky.social
December 7, 2024 at 9:17 PM
Origami tree at the @natural-history.bsky.social
It'll certainly be an opt-in feature in its current form. Forcing a certain deployment pattern will be hard. Otoh, we could automate graceful replica restarts!
November 28, 2024 at 1:40 AM
It'll certainly be an opt-in feature in its current form. Forcing a certain deployment pattern will be hard. Otoh, we could automate graceful replica restarts!
Sonne additional details by @frankmcsherry.bsky.social bsky.app/profile/fran...
Sometimes it's easier to mitigate a problem than solve it: a feature I recently implemented reduces the memory and CPU @materialize.com requires to maintain future updates. Some details why solving the problem is hard and the mitigation easy-ish.
materialize.com/changelog/20...
materialize.com/changelog/20...
Improved performance for temporal filters
Optimization time! We've significantly reduced the cost of using temporal filters in Materialize.
materialize.com
November 27, 2024 at 8:24 PM
Sonne additional details by @frankmcsherry.bsky.social bsky.app/profile/fran...
So, we didn't solve the underlying problem, but mitigated it for many of our workloads.
November 27, 2024 at 6:22 PM
So, we didn't solve the underlying problem, but mitigated it for many of our workloads.
Now, in Materialize time advanced about every second, and the result is high CPU utilization 😿 But: we restart most things every week, so updates beyond a week will never be surfaced! The mitigation we implemented is to drop such updates, and add some safeguards to never produce incorrect results. 🎉
November 27, 2024 at 6:22 PM
Now, in Materialize time advanced about every second, and the result is high CPU utilization 😿 But: we restart most things every week, so updates beyond a week will never be surfaced! The mitigation we implemented is to drop such updates, and add some safeguards to never produce incorrect results. 🎉
Future updates are stashed at the input to indexes, and they'll only act on them once the time has advanced far enough. Because we use Differential Dataflows, the updates are only partially ordered. To extract ready updates, DD scans the data and splits it in ready and pending, ~ in linear time.
November 27, 2024 at 6:22 PM
Future updates are stashed at the input to indexes, and they'll only act on them once the time has advanced far enough. Because we use Differential Dataflows, the updates are only partially ordered. To extract ready updates, DD scans the data and splits it in ready and pending, ~ in linear time.