Lightnews — Scholar-powered news

Reposted by Dewey Dunnington

Andrew Lamb @andrewlamb1111.bsky.social · 14h

Our new thrift parser in the Rust Apache Parquet implementation is a 🎁 that keeps on giving performance wise 🚀 github.com/apache/arrow...

We are also working on a blog post that has a deeper explanation

3 9

Dewey Dunnington @paleolimbot.bsky.social · 1d

Well deserved!

Kyle Barron @kylebarron.dev · 1d

I'm happy to announce I've been promoted to an Apache Arrow Committer, meaning I'm authorized to merge code patches into Arrow repositories!

I join this list of amazing people: arrow.apache.org/committers/

Committers

List of project-management committee (PMC) members and committers on the Apache Arrow project.

arrow.apache.org

3

Reposted by Dewey Dunnington

Andrew Lamb @andrewlamb1111.bsky.social · 1d

Yesterday I learned about the SpatialBench from Sedona github.com/apache/sedon...

Which they based on the tpchgen-rs project from @clflushopt.bsky.social github.com/clflushopt/t...

(BTW I a still looking for some more github watchers on tpchgen-rs so I can get it on homebrew)

1 2

Dewey Dunnington @paleolimbot.bsky.social · 3d

I'm aiming for a weekly Apache Sedona drop...this week's edition: wrangling some uncooperative input data into GeoParquet output using SedonaDB's excellent @arrow.apache.org interop. Bonus: learn about the postage stamps of farm field dotting the prairies!

dewey.dunnington.ca/post/2025/ge...

(Geo)Data cleaning with Arrow and SedonaDB | Dewey Dunnington

Last week I had the pleasure of being a guest on a Cloud Native Geospatial webinar with my colleagues Jia Yu and Matt Forrest where we talked SedonaDB. In the webinar Matt demoed a fantastic blog post...

dewey.dunnington.ca

3

Reposted by Dewey Dunnington

Antoine Pitrou @pitrou.net · 3d

Great news for Arrow (and more work for us 😄).
Also, those work items also implicitly apply to #ApacheParquet. @julien.ledem.net

QuantStack @quantstack.bsky.social · 3d

The Sovereign Tech Agency backs Apache Arrow's Future with QuantStack!

This funding will empower Arrow maintainers at QuantStack, @pitrou.net and @raulcumplido.bsky.social, to advance security, modularity, and long-term sustainability of @arrow.apache.org.

medium.com/@QuantStack/...

The Sovereign Tech Agency Invests in Apache Arrow’s Future with QuantStack

We are thrilled to announce that the Sovereign Tech Fund is providing critical funding to sustain and improve Apache Arrow, an essential…

medium.com

1 2 9

Reposted by Dewey Dunnington

Kyle Barron @kylebarron.dev · 3d

New blog post! kylebarron.dev/blog/fast-py...

This dives into performance pitfalls of Python bindings, why projects like Lonboard are so fast, and why I care about Apache Arrow so much. The post is high level and you don't need to know C/C++/Rust to understand it!

1 4 15

Dewey Dunnington @paleolimbot.bsky.social · 4d

FGB and COG are using the same mechanism that hyparquet, fsspec, objstore, and DuckDB's httpfs are using to read from http (i.e., range requests)

1

Dewey Dunnington @paleolimbot.bsky.social · 5d

Definitely depends what you're up to! People are definitely experimenting with serving features directly from Parquet files for visualization (e.g., medium.com/center-for-c... )

Optimal GeoParquet Partitioning Strategy

My quest for one-size-fits-all format

medium.com

1 2 2

Dewey Dunnington @paleolimbot.bsky.social · 6d

- FGB is great! Lookups with small ranges work well but full scans are much slower (particularly if zipped!)
- SedonaDB, DuckDB, and pyarrow with fsspec can all do partial Parquet reads from http (give geoarrow.org/data a try!)

Data – GeoArrow

geoarrow.org

1 1 1

Dewey Dunnington @paleolimbot.bsky.social · 7d

Interesting...I think of GeoParquet and GeoArrow as tech that allows more people to avoid the cloud (i.e., use your laptop to do things that everyone thought they needed "the cloud" for five years ago). When your 14 GB .fgb files become 2GB GeoParquets, you can fit a whole lot more of them locally!

mdsumner @mdsumner.bsky.social · 7d

"Do things like "GeoParquet" and "GeoArrow" really matter to practitioners who aren't working for a handful of hyper-scaling organizations?". 🤔 First I've heard my employer called that 😆

sgillies.net/2025/09/17/l...

Leaving the GDAL Project Steering Commitee

I've been on the Project Steering Committee (PSC) of the GDAL project for a few years now. It's a hugely significant software project, one that I'm very fond of, and I like to think that I've had an i

sgillies.net

1 4 11

Reposted by Dewey Dunnington

Kyle Walker @kylewalker.bsky.social · 7d

R-Universe by @rOpenSci is an extremely useful project.

My latest R package, {pmtiles}, is not a good fit for CRAN as it bundles a pre-compiled Go library.

With R-Universe, I was able to get installable cross-platform binaries built in hours!

#rstats

walkerke.r-universe....

2 11 40

Dewey Dunnington @paleolimbot.bsky.social · 11d

Last week Apache Sedona announced SedonaDB...and now I finally get to write about it! Here I use SedonaDB to demonstrate the explore/iterate/visualize cycle...SedonaDB's CRS support + @kylebarron.dev's `lonboard` is 👌 for exploring new data!)
dewey.dunnington.ca/post/2025/st...

Stream network traversal with SedonaDB | Dewey Dunnington

Last Wednesday the Apache Sedona project announced SedonaDB. There’s also a great post on the Whereobots blog that has a bit more context, or if you’re like me and you just want to see the code you ca...

dewey.dunnington.ca

2 9

Dewey Dunnington @paleolimbot.bsky.social · 16d

I cannot say enough about DataFusion...in order to build an engine that considers spatial types at every level we needed to customize types, functions, optimizer rules, joins, Parquet pruning, and more. DataFusion not only made this possible but documented even the most obscure bits. So cool!

Andrew Lamb @andrewlamb1111.bsky.social · 16d

"Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen"

Built in Rust with Apache DataFusion

sedona.apache.org/latest/blog/...

Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen - Apache Sedona

Apache Sedona is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark, Apache Flink, and Snowflake, with a set of...

sedona.apache.org

3 25

Reposted by Dewey Dunnington

Andrew Lamb @andrewlamb1111.bsky.social · 16d

"Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen"

Built in Rust with Apache DataFusion

sedona.apache.org/latest/blog/...

Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen - Apache Sedona

Apache Sedona is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark, Apache Flink, and Snowflake, with a set of...

sedona.apache.org

5 15

Dewey Dunnington @paleolimbot.bsky.social · 16d

I'm beyond excited for this! Built on Apache DataFusion and Apache Arrow, a composable single-node analytical database with spatial types as first-class citizens. GeoPandas IO, KNN joins, CRS support, R/Python bindings, and top-notch lazy GeoParquet reads.

sedona.apache.org/latest/blog/...

Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen - Apache Sedona

Apache Sedona is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark, Apache Flink, and Snowflake, with a set of...

sedona.apache.org

1 6

Reposted by Dewey Dunnington

Yutani @yutannihilation-en.bsky.social · 21d

100 stars🎉 Savvy is a niche framework for those who want to use Rust in an R package yet extendr is not their fit, so I didn't expect to see so much stars!

github.com/yutannihilat...

GitHub - yutannihilation/savvy: A simple R extension interface using Rust

A simple R extension interface using Rust. Contribute to yutannihilation/savvy development by creating an account on GitHub.

github.com

2 4

Reposted by Dewey Dunnington

Edzer Pebesma @edzer.mastodon.social.ap.brid.gy · 21d

Spatial Data Science across languages now has a website: https://spatial-data-science.github.io/ Thanks @martinfleis !

Spatial Data Science across Languages

spatial-data-science.github.io

8 6

Reposted by Dewey Dunnington

Bryce @brycemecum.com · Aug 28

Looking forward to attending this! #apachearrow

Apache Arrow @arrow.apache.org · Aug 28

Register for the Apache Arrow Summit Paris 25 (October 2nd) at: www.meetup.com/pydata-paris...
The event is hosted by @pydataparis.bsky.social
We are looking forward to seeing you there and talking about all things Arrow.

Apache Arrow Summit, Thu, Oct 2, 2025, 9:30 AM | Meetup

The day after the PyData Paris conference, we’re excited to host the first-ever Apache Arrow Summit - a gathering dedicated to fostering collaboration and innovation within

www.meetup.com

1 4

Reposted by Dewey Dunnington

Kyle Barron @kylebarron.dev · Aug 22

Published arro3-core v0.6 with wider dtype support for converting numpy ndarrays to Arrow. It now additionally supports:
- datetime64
- timedelta64
- fixed-width strings/bytes
- variable-width strings
- object arrays of str/bytes

And the Python wheel is still just 2.5MB!
github.com/kylebarron/a...

Release py-v0.6.0 · kylebarron/arro3

arro3-core Breaking Changes 🔧 No breaking changes. New features ✨ Expanded dtype support in Array.from_numpy. It now additionally supports datetime64, timedelta64, fixed-width strings and bytes, v...

github.com

1 2 9

Reposted by Dewey Dunnington

Columnar @columnar.tech · Aug 18

In September the @columnar.tech crew are headed to PyData Paris 2025 and the first ever Apache Arrow Summit. The organizer @quantstack.bsky.social is a dedicated supporter of @arrow.apache.org. We’re delighted to be sponsoring the event.

PyData Paris @pydataparis.bsky.social · Aug 18

Welcome to our new sponsor Columnar !

6 9

Reposted by Dewey Dunnington

Jakub Nowosad @jakubnowosad.com · Aug 6

Check out "Scaling the r-spatial ecosystem" by Dewey Dunnington 🌍📦
An exploration of how R’s spatial tools can be used for big(ger) data.

Video: youtu.be/tjNEoIYr_ag?...
Slides: dewey.dunnington.ca/slides/rspat...

#RStats #rspatial #GIS #SpatialData

1 12 16

Reposted by Dewey Dunnington

Yutani @yutannihilation-en.bsky.social · Jul 23

Glad to see another proof that Rust-powered R packages work fine on webR!

For those who are wondering what "savvy" is, it's a framework to use Rust in an R package. #rstats

yutannihilation.github.io/savvy/guide/...

1 2 6

Reposted by Dewey Dunnington

Etienne Bacher @etiennebacher.bsky.social · Jul 23

polars can be used via webR, thanks to the work of @gws.phd both on webR and on the Rust component of polars.

It also uses savvy, developed by @yutannihilation-en.bsky.social‬, to connect Rust and R.

2 6

Reposted by Dewey Dunnington

Etienne Bacher @etiennebacher.bsky.social · Jul 23

#rstats polars 1.0.0 is now available!

polars provides bindings to the eponymous Rust library, a very efficient dataframe library.

This is the outcome of a 1-year long rewrite led by @eitsupi to refactor the internals and match the API of Python polars.

pola-rs.github.io/r-polars/

Details ⬇️

Polars R Package

pola-rs.github.io

4 12 36

Dewey Dunnington @paleolimbot.bsky.social · Jul 22

A new version of @arrow.apache.org is out! I'm personally excited about the Parquet support for Geometry/Geography but there's a lot of other great features in this release...check out the release post for all the details! arrow.apache.org/blog/2025/07...

5 13