@stanislavkozlovski.bsky.social
Data retention in disk/memory is decoupled from whether the message was consumed or not. Unlike certain message queues, this ensures availability is unaffected (can’t run out of disk/memory) from how consumers behave
November 18, 2025 at 3:38 PM
Data retention in disk/memory is decoupled from whether the message was consumed or not. Unlike certain message queues, this ensures availability is unaffected (can’t run out of disk/memory) from how consumers behave
A leader/follower replication model does not require strong distributed consensus between too many nodes to serve traffic.
November 18, 2025 at 3:38 PM
A leader/follower replication model does not require strong distributed consensus between too many nodes to serve traffic.
Partitions act as a sharding mechanism. A single partition usually does not push more than a few MB/s.
November 18, 2025 at 3:38 PM
Partitions act as a sharding mechanism. A single partition usually does not push more than a few MB/s.
A Kafka cluster scales by horizontally adding more nodes (called brokers).
November 18, 2025 at 3:38 PM
A Kafka cluster scales by horizontally adding more nodes (called brokers).
The append-only nature of the log (i.e no updates/deletes) means that reads do not require a quorum to serve fresh data.
There are also no locks between concurrent reads or writes, hence no contention.
There are also no locks between concurrent reads or writes, hence no contention.
November 18, 2025 at 3:38 PM
The append-only nature of the log (i.e no updates/deletes) means that reads do not require a quorum to serve fresh data.
There are also no locks between concurrent reads or writes, hence no contention.
There are also no locks between concurrent reads or writes, hence no contention.
Kafka heavily leverages the OS file system cache (page cache) to serve reads fast from memory
November 18, 2025 at 3:38 PM
Kafka heavily leverages the OS file system cache (page cache) to serve reads fast from memory
Physical disk writes are heavily batched on the server-side. The fact that Kafka writes linear blocks to disk and fsync is not used allows the OS to batch up data in memory (page cache), coalescing the data into larger IO operations
November 18, 2025 at 3:38 PM
Physical disk writes are heavily batched on the server-side. The fact that Kafka writes linear blocks to disk and fsync is not used allows the OS to batch up data in memory (page cache), coalescing the data into larger IO operations
- since the synchronous write path is literally writing to memory. Producers can also be configured to not block on this replication consensus when writing (acks=1 vs acks=all).
November 18, 2025 at 3:38 PM
- since the synchronous write path is literally writing to memory. Producers can also be configured to not block on this replication consensus when writing (acks=1 vs acks=all).
Writes by default require consensus that every follower replica has acknowledged the write before returning, but Kafka works with fsync off by default and therefore writes to disk asynchronously, without blocking on the physical disk write. This provides near memory-like performance when well-tuned-
November 18, 2025 at 3:38 PM
Writes by default require consensus that every follower replica has acknowledged the write before returning, but Kafka works with fsync off by default and therefore writes to disk asynchronously, without blocking on the physical disk write. This provides near memory-like performance when well-tuned-
The broker does not touch the data, its a dumb pipe - messages aren’t even validated for their schema, hence compressed data isn’t decompressed on the server side
November 18, 2025 at 3:38 PM
The broker does not touch the data, its a dumb pipe - messages aren’t even validated for their schema, hence compressed data isn’t decompressed on the server side
Compression is usually also done on the client, and these larger batches compress much more favorably. This leads to larger network packets, larger sequential disk operations, contiguous memory blocks
November 18, 2025 at 3:38 PM
Compression is usually also done on the client, and these larger batches compress much more favorably. This leads to larger network packets, larger sequential disk operations, contiguous memory blocks
Writes are batched on the client-side. Producers batch multiple messages into one "record-batch", hence increasing throughput per request.
November 18, 2025 at 3:38 PM
Writes are batched on the client-side. Producers batch multiple messages into one "record-batch", hence increasing throughput per request.
The system supports two main very simple and performant operations:
- Produce, which appends to the end of the log
- Consume, which reads sequentially starting from any particular offset
- Produce, which appends to the end of the log
- Consume, which reads sequentially starting from any particular offset
November 18, 2025 at 3:38 PM
The system supports two main very simple and performant operations:
- Produce, which appends to the end of the log
- Consume, which reads sequentially starting from any particular offset
- Produce, which appends to the end of the log
- Consume, which reads sequentially starting from any particular offset
The Android feels choppy/flow and the touch also doesn't feel super responsive
November 11, 2025 at 8:57 PM
The Android feels choppy/flow and the touch also doesn't feel super responsive
Daylight is amazing. I'm really looking forward to a v2 (whenever it comes) that's a bit more refined
November 11, 2025 at 6:51 PM
Daylight is amazing. I'm really looking forward to a v2 (whenever it comes) that's a bit more refined
I mean, yeah - it's not a technical piece, it's a financial article.
When I hear the Kafka community, I think about engineering talk more than anything else
When I hear the Kafka community, I think about engineering talk more than anything else
November 11, 2025 at 6:49 PM
I mean, yeah - it's not a technical piece, it's a financial article.
When I hear the Kafka community, I think about engineering talk more than anything else
When I hear the Kafka community, I think about engineering talk more than anything else
With greater competition and commercialization, it seems bound to happen.
I'm of the opinion we're bound to see consolidation in the space soon, because there's too many companies chasing too little of a market: bigdata.2minutestreaming.com/p/event-stre...
I'm of the opinion we're bound to see consolidation in the space soon, because there's too many companies chasing too little of a market: bigdata.2minutestreaming.com/p/event-stre...
November 10, 2025 at 3:34 PM
With greater competition and commercialization, it seems bound to happen.
I'm of the opinion we're bound to see consolidation in the space soon, because there's too many companies chasing too little of a market: bigdata.2minutestreaming.com/p/event-stre...
I'm of the opinion we're bound to see consolidation in the space soon, because there's too many companies chasing too little of a market: bigdata.2minutestreaming.com/p/event-stre...
Then I don't see why a relational database can't maintain a pending queue too. Just process the lowest-id job
November 8, 2025 at 10:45 PM
Then I don't see why a relational database can't maintain a pending queue too. Just process the lowest-id job
What are you referring to?
My intuition is rather that discussion has somewhat died down, in general.
My intuition is rather that discussion has somewhat died down, in general.
November 8, 2025 at 10:44 PM
What are you referring to?
My intuition is rather that discussion has somewhat died down, in general.
My intuition is rather that discussion has somewhat died down, in general.
re: fairness - I admit I'm not as familiar with. Can you teach me how some of these systems ensure fairness? Do they refuse to give messages to performant workers and hold them off for slower ones?
November 8, 2025 at 2:21 PM
re: fairness - I admit I'm not as familiar with. Can you teach me how some of these systems ensure fairness? Do they refuse to give messages to performant workers and hold them off for slower ones?
There is some fundamental misunderstanding here.
pgmq is not based on top of SQS. It only provides API parity with it. No messages actually go to SQS...
Latency-wise, my tests showed single-digit write and read. 99% use cases don't need less.
pgmq is not based on top of SQS. It only provides API parity with it. No messages actually go to SQS...
Latency-wise, my tests showed single-digit write and read. 99% use cases don't need less.
November 8, 2025 at 2:21 PM
There is some fundamental misunderstanding here.
pgmq is not based on top of SQS. It only provides API parity with it. No messages actually go to SQS...
Latency-wise, my tests showed single-digit write and read. 99% use cases don't need less.
pgmq is not based on top of SQS. It only provides API parity with it. No messages actually go to SQS...
Latency-wise, my tests showed single-digit write and read. 99% use cases don't need less.
I really don't know. I haven't evaluated it yet.
November 8, 2025 at 2:18 PM
I really don't know. I haven't evaluated it yet.