Docs: github.com/Snowflake-La...

No system hits the sweet spot of allowing for extensibility while maintaining systems safety. It would be nice if there was a standard plugin API (think POSIX) that allows compatibility across systems.

Thanks to @marcoslot.com + @daveandersen.bsky.social for their collaboration on this project

Thanks Gunnar!

We generate regular position delete files for merge-on-read, so any Iceberg query engine can read them. Equality deletes would be more CDC friendly, but not supported in most engines.

We have some secret sauce around how we track/know positions, but being Postgres helps a lot there.

Would be cool if Iceberg/Parquet had support for storing JSON as vectors.

Generally by unwinding the JSON in the insert..select that processes the raw log files. JSON is broken down into columns by default, though nested JSON remains as jsonb. You can either store that directly (stored as string), unwind it manually, or convert to a composite type (stored as struct)