Lightnews — Scholar-powered news

Reposted by mdsumner

econmaett @econmaett.github.io · 18h

Ross Ihaka: The Open-Hearted Data Hero
#rstats
www.historyofdatascience.com/ross-ihaka-t...

“R changed my opinion of humanity to some extent, to see how people are really willing to freely give of themselves and produce something larger than themselves without any thought of personal glory.”

mdsumner @mdsumner.bsky.social · 21h

how's your day going

#rstats

paste0(sample(c(letters, LETTERS, 0:9,
strsplit("!@#$%^&*()_+=-~", "")[[1]]), 56),
collapse = "")

2 4

Reposted by mdsumner

Mike Mahoney @mikemahoney218.com · 6d

my recent focus has been on a lot of point time series data -- I _think_ someone told me zarr handles this use case well enough, but honestly I've only seriously played with parquet so far

1 1

Reposted by mdsumner

Lawrence Vulis @lvulis.bsky.social · 6d

Oh huh. Like trajectories? I'd keep in mind what you're doing up/down as well and how that fits into the workflow. So what packages are loading the parquets etc because that ends up being a huge part of the decisionmaking for me

1 1

mdsumner @mdsumner.bsky.social · 1d

Oh great point ! is actually exactly where tabular spatial breaks down, trajectory data is an excellent crossover that doesn't quite fit either world, it's mostly tabular but ordered within groups (and is it points, segments, line strings or all the above, yes).

Argo long sits in this uneasy place

1

mdsumner @mdsumner.bsky.social · 1d

This is not parquet, but pretty essential to see where xarray and Zarr are going (raster yes, but not just raster)

youtu.be/I-NHCuLhRjY?...

mdsumner @mdsumner.bsky.social · 1d

It won't be long until Icechunk is used to virtualize Parquet stores, referencing compressed chunks by byte ranges, and stored in their mysterious Rust Zarrtabase

1

mdsumner @mdsumner.bsky.social · 1d

Neither is quite the superset, but both can use the other and this is not a hierarchy rather a thicket of experiments

1

mdsumner @mdsumner.bsky.social · 1d

Also, you already know will the difference imo because it's manifest in R, arrays (missing "labels) vs data frames

But, it's those crossover areas where they bleed into one another (nested parquet like json arrays, compressed partition chunks like in arrow) is where the blur is.

1 1

mdsumner @mdsumner.bsky.social · 1d

And this one:

earthmover.io/blog/what-is...

(The Earthmover stance that "labelled arrays are inefficient as fully materialized tables" is a bit disingenuous ... obvious actually, I wonder if they will review better now xarray has coordinate indexes)

Fundamentals: What Is Zarr? A Cloud-Native Format for Tensor Data - Earthmover

What Zarr is, and how it enables fast, scalable access to multidimensional array data in the cloud.

earthmover.io

2

Reposted by mdsumner

JimmyC @jimmyc42.bsky.social · 6d

Have you seen: earthmover.io/blog/fundame...

Fundamentals: What is Cloud-Optimized Scientific Data?

What cloud-optimized data really means, and how Zarr and Icechunk enable fast access to massive scientific datasets in cloud object storage.

earthmover.io

1 1 4

mdsumner @mdsumner.bsky.social · 1d

"hit", I meant "gut"

Reposted by mdsumner

Mike Mahoney @mikemahoney218.com · 6d

hey spatial folks: if anyone has a great "parquet versus zarr" write-up, let me know. I'm never going to write a serializer but I'm really interested in the performance impacts of format for cloud downloads 👀

#geosky

3 4 8

mdsumner @mdsumner.bsky.social · 1d

Have you seen kerchunk store virtual Zarr in Parquet? (Also, intake models netcdf by nesting array metadata in Parquet

(Comparing Zarr to parq s a category error imo, like vector cubes: there's no cube it's just a relation to not duplicate structures, stored via arrays rather than latent joins)

1 1

mdsumner @mdsumner.bsky.social · 1d

They're real different... don't do the same things ... Parquet is a table, until it's a nested table (then it's more like nested json, which is where the array folks pounce)

The array folks think Zarr will take over all table tech... (I think arrow will hit the Zarr core and become its foundation)

2 1

Reposted by mdsumner

Ilja @fubits.dev · 4d

ah yeah, absolutely adding `zarr` to the watch-list! bsky.app/profile/mike...

Mike Mahoney @mikemahoney218.com · 6d

hey spatial folks: if anyone has a great "parquet versus zarr" write-up, let me know. I'm never going to write a serializer but I'm really interested in the performance impacts of format for cloud downloads 👀

#geosky

1

Reposted by mdsumner

olliniemitalo.bsky.social @olliniemitalo.bsky.social · 2d

Zarr in the Browser — Fast, Flexible, and Surprisingly Powerful for Big Geo Data by Tobias Ferreira, Jul 1, 2025 medium.com/p/zarr-in-th...

1 1

Reposted by mdsumner

Mike O'Brien @obrien.page · 7d

(the read-write end -- the server is well beyond this and awesome by itself)

1

Reposted by mdsumner

Mike O'Brien @obrien.page · 7d

I rescind this question now that I'm getting further into the package 🫣

1 1

mdsumner @mdsumner.bsky.social · 1d

star wars won't work

Reposted by mdsumner

John Ross @johnrossmd.bsky.social · 1d

Pleased to report that this 88-year-old fella still has the juice

Shadow Ticket, a new novel by Thomas Pynchon

2 7

Reposted by mdsumner

energy9086.bsky.social @energy9086.bsky.social · 1d

28 240 690

mdsumner @mdsumner.bsky.social · 1d

#python what dataframe framework will hand over a column without it always being a (useless) Series?

gawd, it feels like being stuck in {zoo}

Reposted by mdsumner

Robert Simmon @rsimmon.bsky.social · 2d

Any chance you could point to an example?

1 1

mdsumner @mdsumner.bsky.social · 1d

scratching the surface here: gist.github.com/mdsumner/f25...

the real power I find is that it's so modular, we can fix a huge list of problems and read from urls with vrt:// / vsicurl, and warping can burn pixels into any grid we want so the combination is hugely various

2