Lightnews — Scholar-powered news

Dmitriy Ryaboy @squarecog.bsky.social · 7d

Right, it's all about the ecosystem. Writers are always going to be more conservative than readers, rightfully so. This f3 idea is essentially about letting writers adopt new stuff without worrying too much about older gen readers (once older gen can read this sort of thing, another decade later).

2

Dmitriy Ryaboy @squarecog.bsky.social · 7d

bsky.app/profile/andy...

Andy Pavlo @andypavlo.bsky.social · 14d

One problem with Parquet is many implementations are not updated when the official spec improves. Everyone just uses the lowest version feature set. That means if Parquet adds a better data encoding scheme and a file uses it, many common reader libraries won't be able retrieve the data.

Survey of the features used in public Parquet files.

1 1

Dmitriy Ryaboy @squarecog.bsky.social · 7d

But anyway the point is not whether rle is useful, but if there is a world where parquet format improvements introduced since like 2018 get adopted, and more useful encodings can be propagated.

1

Dmitriy Ryaboy @squarecog.bsky.social · 7d

RLE+delta allows filter pushdowns to work without decompressing. If you have repeated strings and sort, dict encode, and rle+delta, even regex searches become blazing fast. Parquet enables this, but who implements it?

1

Dmitriy Ryaboy @squarecog.bsky.social · 8d

To be fair, you would not require it. An implementor would only do this if they want to future proof, and are ok with the whole executable data file thing. Otherwise, same as now: implement the reader for every encoding.
It's painful how little even basic RLE is being used in the wild :(

1

Dmitriy Ryaboy @squarecog.bsky.social · 14d

I had the same 2 thoughts in the same sequence :)

1 2

Dmitriy Ryaboy @squarecog.bsky.social · 14d

Do you think there's anything blocking parquet from adopting the same wasm reader approach to unlock new encodings and other schemes?

1 2

Dmitriy Ryaboy @squarecog.bsky.social · 14d

This is a pretty intriguing idea for future proofing file formats.
It does assume wasm is future proof, of course, but that feels like a safer bet than "assume readers are updated"

Andy Pavlo @andypavlo.bsky.social · 14d

Our F3 files embed small WASM programs to decode data. If somebody creates a new encoding and the DBMS does not have native impl, it can still read data using WASM passing Arrow buffers. Our experiments show WASM is 15-20% slower than native. We use @spiraldb.com's Vortex encoding impls.

Overview of F3's decoding pipeline with WASM support.

1 4

Dmitriy Ryaboy @squarecog.bsky.social · 15d

If you love this sort of thing, read up on C-store, which introduced this idea in 2005 and commercialized it in Vertica. Stonebraker, Sam Madden, Daniel Abadi.
Parquet was also partially inspired by Vertica (and Google's Dremel, and PaX by Natassa Ailamaki et al) :-).

DuckDB @duckdb.org · 16d

Are you streaming into your Lakehouse?

Traditional formats suffered with the “many small files” problem — OLAP engines merge them reactively with long jobs. ⏳

DuckLake takes a proactive path: Data Inlining + async flush to parquet while always keeping data queryable ⚡

1 2

Dmitriy Ryaboy @squarecog.bsky.social · 19d

The original Yo app.

1 2

Dmitriy Ryaboy @squarecog.bsky.social · 23d

ML is just applied stats.
Stats is just applied algebra.
LLM is just ML backward and with an extra L.

Dmitriy Ryaboy @squarecog.bsky.social · 25d

The obvious reaction here is to shift at least some of the hiring out of the country to get access to the talent. The obvious counter reaction is to tax payments and wages to foreign employees and contractors. Which will also provoke a reaction. And none of this makes the US stronger or smarter.

4

Dmitriy Ryaboy @squarecog.bsky.social · 25d

About a decade late with this, but:
Someone should have started a social media ad agency called Twaddle.

Dmitriy Ryaboy @squarecog.bsky.social · 25d

Ask Ketan, I've been trying to find a good excuse to get my teams to use Flyte for half a decade now 😆

1

Dmitriy Ryaboy @squarecog.bsky.social · 27d

It's tempting to take shortcuts that give you speed today by mortgaging speed tomorrow.

Trouble is, today is yesterday's tomorrow.

1

Dmitriy Ryaboy @squarecog.bsky.social · 28d

Thanks for the reference, hadn't seen that!
Are these all one-shotting or doing an agentic workflow to explore before formulating final answer?

1

Dmitriy Ryaboy @squarecog.bsky.social · 29d

I tried 2 different english to insights sql llm agents from reputable vendors in the past week. Data analyst jobs remain safe.
Firmly in the toy category for now.

1 1 13

Dmitriy Ryaboy @squarecog.bsky.social · Sep 16

Happened to be by the Cloudera building in the south bay earlier. Checked LinkedIn and discovered I have literally 0 1st degree connections who work there now. Not unexpected, I guess, but, man... betwen hnwx and cldr I used to know like 100s of folks there

1 5

Dmitriy Ryaboy @squarecog.bsky.social · Sep 15

Heck of a vote of confidence from LinkedIn for Flyte: www.linkedin.com/blog/enginee...

OpenConnect: LinkedIn's Next-Generation AI Pipeline Ecosystem

www.linkedin.com

1 1 3

Dmitriy Ryaboy @squarecog.bsky.social · Sep 11

Also, everyone trips at least once on average of ratios vs ratio of sums (which becomes obvious once you describe them as unweighted vs weighted means).

1

Dmitriy Ryaboy @squarecog.bsky.social · Sep 11

It shows up in different ways in different places. The most basic being, you don't know if the rario moved cause numerator went up or denominator went down. Correct course of action is often different depending on which!

1 1

Dmitriy Ryaboy @squarecog.bsky.social · Sep 10

About once every two years I have cause to re-learn a very important data lesson: never, ever, trust analysis based on ratio metrics.

1 2

Dmitriy Ryaboy @squarecog.bsky.social · Aug 30

Trying and failing to make the page edits look right? Is offline access lackluster? Tired of AI upsell as a replacement for poor search quality?

You might be suffering from Notion sickness.

5

Dmitriy Ryaboy @squarecog.bsky.social · Aug 26

Phone book, noun: an ebook you read on your phone.

2

Dmitriy Ryaboy @squarecog.bsky.social · Aug 14

Looking up latin phrases on Google results in an AI response in French a good % of the time. Fortunately, my French is slightly better than my latin.

1