Lightnews — Scholar-powered news

Jack Vanlightly @vanlightly.bsky.social · 1d

Which will be proprietary platform stuff I assume, so not going into the Iceberg spec? Is there any chance of caching layers also being open and standardised?

Jack Vanlightly @vanlightly.bsky.social · 1d

Why don’t Iceberg or Delta Lake have secondary indexes?
Because analytics workloads and OLTP workloads optimize for opposite I/O patterns.

See my dive into data layout, pruning, and what “indexing” really means in open table formats: jack-vanlightly.com/blog/2025/10...

Beyond Indexes: How Open Table Formats Optimize Query Performance — Jack Vanlightly

My career in data started as a SQL Server performance specialist, which meant I was deep into the nuances of indexes, locking and blocking, execution plan analysis and query design. These days I’m mor...

jack-vanlightly.com

2 2 15

Jack Vanlightly @vanlightly.bsky.social · Sep 2

New deep dive: Understanding Apache Fluss

I spent August reverse-engineering Fluss, Alibaba’s new table storage engine for Flink (partially forked from Kafka). This post covers its architecture, tiering, and how it tackles changelogs & low-latency state.

jack-vanlightly.com/blog/2025/9/...

Understanding Apache Fluss — Jack Vanlightly

This is a data system internals blog post. So if you enjoyed my table formats internals blog posts , or writing on Apache Kafka internals or Apache BookKeeper internals , you might enjoy thi...

jack-vanlightly.com

2 15

Jack Vanlightly @vanlightly.bsky.social · Aug 21

New blog post: A Conceptual Model for Storage Unification.

The post defines what storage unification means, defines terminology and evaluates different building blocks and approaches to doing it.

jack-vanlightly.com/blog/2025/8/...

A Conceptual Model for Storage Unification — Jack Vanlightly

Object storage is taking over more of the data stack, but low-latency systems still need separate hot-data storage. Storage unification is about presenting these heterogeneous storage systems and form...

jack-vanlightly.com

2 8

Jack Vanlightly @vanlightly.bsky.social · Jul 28

In a future of autonomous AI agents, we can't limit ourselves to error prevention and error detection, we must also include remediation.

jack-vanlightly.com/blog/2025/7/...

Remediation: What happens after AI goes wrong? — Jack Vanlightly

If you’re following the world of AI right now, no doubt you saw Jason Lemkin’s post on social media reporting how Replit’s AI deleted his production database , despite it being told not to touch an...

jack-vanlightly.com

2

Jack Vanlightly @vanlightly.bsky.social · Jul 22

Ha! The Blondlot example is fascinating. Sometimes you can fail so spectacularly that they have to invent new math controls just to prevent future people from making a mistake as bad as yours. That's a special kind of immortality.

1 2

Jack Vanlightly @vanlightly.bsky.social · Jul 22

Science moves slowly because wrong theories waste decades. Engineering is careful because failures kill people. Software moves fast because mistakes are cheap, the expensive error isn't making the wrong choice, it's taking too long to make any choice. jack-vanlightly.com/blog/2025/7/...

The Cost of Being Wrong — Jack Vanlightly

A recent LinkedIn post by Nick Lebesis caught my attention with this brutal take on the difference between good startup founders and coward startup founders. I recommend you read the entire thing ...

jack-vanlightly.com

1 4

Jack Vanlightly @vanlightly.bsky.social · Jul 16

But no, I was not offended 😄 Keep on writing it how you see it!

Jack Vanlightly @vanlightly.bsky.social · Jul 16

He could easily have framed it in a more positive way. But either way, I don't think it matters too much, your readers are intelligent and can benefit from both, and see the criticism from a more positive angle (picking the bits they like from both).

1

Jack Vanlightly @vanlightly.bsky.social · Jul 16

Your post was valid from that scale. Winston's response took issue with it because he's been thinking very deeply at the macro-scale of power systems that run our entire planet. And I appreciate his views there. But, I think it was an unfair take out and needlessly combative.

1

Jack Vanlightly @vanlightly.bsky.social · Jul 16

I think it's a matter of scale. For me, your post is from the position where you are at day-to-day, your subjective experience, helping run a company, pushing back on some of the BS you see in the industry (which we love!), all inextricably linked to your past.

1 2

Jack Vanlightly @vanlightly.bsky.social · Jul 15

Where does reliability begin, and where does it end? In distributed business architectures, the answer is responsibility boundaries. New post: jack-vanlightly.com/blog/2025/7/...

Responsibility Boundaries in the Coordinated Progress model — Jack Vanlightly

Building on my previous work on the Coordinated Progress model, this post examines how reliable triggers not only initiate work but also establish responsibility boundaries . Where a reliable tri...

jack-vanlightly.com

5 12

Jack Vanlightly @vanlightly.bsky.social · Jul 3

Next time, I'll agree it's Wednesday 😆

2

Jack Vanlightly @vanlightly.bsky.social · Jul 3

ChatGPT thought it was Tuesday, so I made fun of it and it admitted it was Wednesday. So I made fun of it again, and it admitted it was...Wednesday. But sure, AI agents are gonna steal my job 🤔

1 3

Jack Vanlightly @vanlightly.bsky.social · Jul 1

Like how to write an if statement or loop in bash 😄 I swear my brain is incapable of remembering that.

3 9

Jack Vanlightly @vanlightly.bsky.social · Jun 24

It really makes me question how ready it is for autonomous agents. I'm still on the "I'll believe it when I see it" camp for AI agents.

1 6

Jack Vanlightly @vanlightly.bsky.social · Jun 24

ChatGPT has hallucinated so many times for me today. It's invented scientific terms that don't exist, has been quite liberal with plausible answers based on what sounds reasonable, but without any real world justification. When challenged, it admits it's mistake.

1 2

Jack Vanlightly @vanlightly.bsky.social · Jun 20

At this point, I can't tell if coffee makes me feel better in the morning because I am in withdrawal, or it actually picks me up.

2 3

Jack Vanlightly @vanlightly.bsky.social · Jun 13

My musical evolution continues, discovered deep hypnotic drone music today. No drugs required 😄 The Hypnus Records label is great.

1 3

Jack Vanlightly @vanlightly.bsky.social · Jun 12

The abstraction seems to make sense to people. And it isn't complicated either which is nice.

Jack Vanlightly @vanlightly.bsky.social · Jun 11

How to reliably distribute work across microservices, stream processors, durable execution, event-driven, orchestration and now AI agents?

Coordinated Progress is a 4 part series that explores the common structure behind reliable distributed systems.

jack-vanlightly.com/blog/2025/6/...

Coordinated Progress – Part 1 – Seeing the System: The Graph — Jack Vanlightly

At some point, we’ve all sat in an architecture meeting where someone asks, “ Should this be an event? An RPC? A queue? ”, or “ How do we tie this process together across our microservices? Should it ...

jack-vanlightly.com

3 8 33

Jack Vanlightly @vanlightly.bsky.social · Jun 9

I took a break from social media and my blog for a couple of months. ND burnout. But I'm tentatively back, probably just to post my writing here for now. HOTDS is on pause. Getting back to writing is therapeutic though. I'll post something this week that I've been working on.

7

Jack Vanlightly @vanlightly.bsky.social · Jun 9

Agree, it's not the Hz but any kind of calming sound like pouring rain, or these slow atmospheric noises is what does it for me. It quietens my brain. In fact, since I discovered this, I haven't listened to music once. I only listen to these calming tracks now.

Jack Vanlightly @vanlightly.bsky.social · Apr 4

My pleasure 😄

Jack Vanlightly @vanlightly.bsky.social · Apr 4

Another Humans of the Data Sphere is out, with issue 10! In this issue people are talking fsyncs, tips for running ClickHouse at scale, the problems with MCP and more. Plus I dig up a classic paper from 1962. www.hotds.dev/p/humans-of-...

Humans of the Data Sphere Issue #10 April 4th 2025

Your biweekly dose of insights, observations, commentary and opinions from interesting people from the world of databases, AI, streaming, distributed systems and the data engineering/analytics space.

www.hotds.dev

1 4 5