Lightnews — Scholar-powered news

Nikhil Benesch @benesch.bsky.social · Jul 22

They’re calling it our most boring feature yet.

boring(n): mundane; works exactly as expected; the highest praise you can give a database

turbopuffer @turbopuffer.bsky.social · Jul 22

by popular demand, now introducing one-dimensional vectors 🎈

3

Nikhil Benesch @benesch.bsky.social · Jun 10

tpuf Python client went async today! 🐡 ❤️ 🐍

Async client perf slightly edges out sync client perf under heavy query load.

github.com/turbopuffer/...

4

Nikhil Benesch @benesch.bsky.social · May 14

Big day today at turbopuffer HQ.

turbopuffer @turbopuffer.bsky.social · May 14

turbopuffer is generally available. petabytes in prod.

turbopuffer.com/

1 8

Reposted by Nikhil Benesch

turbopuffer @turbopuffer.bsky.social · Mar 7

when we said "coming soon" we really meant it

now puffin' in aws-us-east-1 and aws-eu-central-1

1 3

Nikhil Benesch @benesch.bsky.social · Mar 7

I did! Couldn't pass up the opportunity to work with this team!

1

Nikhil Benesch @benesch.bsky.social · Mar 7

Things move fast at turbopuffer. Now puffin' in aws/us-east-1 and aws/eu-central-1 too.

1 3

Nikhil Benesch @benesch.bsky.social · Mar 6

I've been dreaming about conditional writes on S3 for years. I couldn't have asked for a better way to celebrate than getting to ship tpuf on AWS. 🐡

turbopuffer @turbopuffer.bsky.social · Mar 6

now available: turbopuffer AWS regions ☁️

come find us in us-west-2 and ap-southeast-2

1 13

Nikhil Benesch @benesch.bsky.social · Feb 27

C4A: cloud.google.com/blog/product...

Try C4A, the first Google Axion Processor | Google Cloud Blog

The custom Arm-based processor is designed for general-purpose workloads like web and app servers, databases, analytics, CPU-based AI, and more.

cloud.google.com

1 1

Nikhil Benesch @benesch.bsky.social · Feb 26

It’s a rare day that my love of going to battle with build systems pays off like this. Kudos to GCP for a very impressive new SKU. 🐡💨

turbopuffer @turbopuffer.bsky.social · Feb 26

GCP released a new ARM machine type with NVMe SSDs recently

70% increase in end-to-end production indexing throughput at 20-33% lower machine cost!

12

Nikhil Benesch @benesch.bsky.social · Feb 6

Couldn't be more excited to be joining the team at @turbopuffer.bsky.social. 🐡💨

turbopuffer @turbopuffer.bsky.social · Feb 6

welcoming two excellent additions to team tpuf this week: @arash11gt as a customer engineer, and @nikhilbenesch to the DB team from pasts at Materialize & Cockroach.

we're locking in a p999 eng team to build the best search engine for scale.

17

Nikhil Benesch @benesch.bsky.social · Dec 30

Just catching up on my NULL BITMAPS and this is easily the best intuition for write skew I've ever seen described. The analogy to merge skew in a codebase is genius.

Justin @jaffray.bsky.social · Dec 23

this week on NULL BITMAP, intuition for snapshot isolation: it’s like git buttondown.com/jaffray/arch...

Git Workflow is Snapshot-Isolated

We are apparently continuing our series of posts borne from thinking about Amazon DSQL. Today we are talking about skew. Amy and Bill are two developers...

buttondown.com

9

Nikhil Benesch @benesch.bsky.social · Dec 19

Well well well: www.crunchydata.com/blog/pg_incr...

Incremental pipelines come to Postgres via Crunchy Data! This is like "dbt incremental", not true incremental view maintenance like @materialize.com or Snowflake's dynamic tables, but it's a neat step towards IVM.

3 21

Nikhil Benesch @benesch.bsky.social · Dec 19

Of course! Will be very excited to give the new DynamoDB-free SlateDB a spin.

1

Reposted by Nikhil Benesch

Phil Eaton @eatonphil.bsky.social · Dec 12

The 6th (and last of 2024!) NYC Systems talks are next Thursday! We've got @jaronoff.com of Omlet and @benesch.bsky.social of Materialize speaking. :)

nycsystems.xyz/december-202...

2 20

Nikhil Benesch @benesch.bsky.social · Dec 11

Same! I’ve been using Amethyst for over ten years at this point. When I discovered the project in 2013 I never expected that ianyh would still be lovingly maintaining it a decade later. One of my most loved pieces of software, for sure.

1 2

Nikhil Benesch @benesch.bsky.social · Dec 11

tl;dr the 10x TPS claim actually *is* based on a small but novel optimization inside of S3! Table buckets understand Iceberg naming conventions and adjust S3's rate limiting so that you start with budget for 55k/35k GETs/PUTs per sec rather than the general purpose default of 5.5k/3.5k.

1 7

Nikhil Benesch @benesch.bsky.social · Dec 11

It just occurred to me that while the 3x query performance claim is based on a comparison to uncomplicated tables, I hadn't actually seen the evidence for the 10x TPS claim.

After some digging I just found the explanation in the re:Invent talk (start at 14:24): youtu.be/1U7yX4HTLCI?...

AWS re:Invent 2024 - [NEW LAUNCH] Store tabular data at scale with Amazon S3 Tables (STG367-NEW)

YouTube video by AWS Events

youtu.be

1 3

Nikhil Benesch @benesch.bsky.social · Dec 11

Exciting! I'm so here for it. 👀

Nikhil Benesch @benesch.bsky.social · Dec 10

Disclaimer: I'm not involved with DuckDB, ClickHouse, or Iceberg, so take this all with a grain of salt.

1 1

Nikhil Benesch @benesch.bsky.social · Dec 10

Rust might be feasible! In fact I think DuckDB's Delta extension is based on a Rust library. And ClickHouse is starting to integrate some Rust too: clickhouse.com/blog/more-th...

Sadly Rust's Iceberg library is still relatively immature (e.g. no support for writes: github.com/apache/icebe...).

More Than 2x Faster Hashing in ClickHouse Using Rust

Rust’s rich type system and ownership model guarantee memory-safety and thread-safety. There are a fairly large number of useful libraries written on it, so we considered using them in ClickHouse.

clickhouse.com

1 4

Nikhil Benesch @benesch.bsky.social · Dec 10

Calling Go code from another language is hard enough that' it's rarely done—especially in high performance contexts. (🌶️ alert, but fasterthanli.me/articles/lie... covers this—see "the only good boundary with Go is a network boundary.")

Lies we tell ourselves to keep using Golang

In the two years since I've posted I want off Mr Golang's Wild Ride, it's made the rounds time and time again, on Reddit, on Lobste.rs, on HackerNews, and elsewhere. And every time, it elicits the ...

fasterthanli.me

1 4

Nikhil Benesch @benesch.bsky.social · Dec 10

Nice, looking forward to it!

Nikhil Benesch @benesch.bsky.social · Dec 10

I’m not 100% sure on the chronology but I believe DuckDB originated many of these pragmatic SQL UX improvements: duckdb.org/2022/05/04/f...

Friendlier SQL with DuckDB

DuckDB offers several extensions to the SQL syntax. For a full list of these features, see the Friendly SQL documentation page.

duckdb.org

2

Nikhil Benesch @benesch.bsky.social · Dec 10

Keep an eye on github.com/apache/icebe...! AFAIU the lack of a C++ Iceberg library is what’s been holding back full Iceberg support in DuckDB and ClickHouse.

GitHub - apache/iceberg-cpp: Apache Iceberg C++

Apache Iceberg C++. Contribute to apache/iceberg-cpp development by creating an account on GitHub.

github.com

1 1 3

Nikhil Benesch @benesch.bsky.social · Dec 10

@chris.blue you might be interested in this. Something like this fits in very neatly to your views on the commoditization of the PostgreSQL dialect/protocol (materializedview.io/p/databases-...).

Databases Are Commodities. Now What?

How will vendors differentiate when databases are commoditized? I've got three ideas.

materializedview.io

1 4