Lightnews — Scholar-powered news

Dewey Dunnington @paleolimbot.bsky.social · 23h

I'm aiming for a weekly Apache Sedona drop...this week's edition: wrangling some uncooperative input data into GeoParquet output using SedonaDB's excellent @arrow.apache.org interop. Bonus: learn about the postage stamps of farm field dotting the prairies!

dewey.dunnington.ca/post/2025/ge...

(Geo)Data cleaning with Arrow and SedonaDB | Dewey Dunnington

Last week I had the pleasure of being a guest on a Cloud Native Geospatial webinar with my colleagues Jia Yu and Matt Forrest where we talked SedonaDB. In the webinar Matt demoed a fantastic blog post...

dewey.dunnington.ca

3

Reposted by Dewey Dunnington

Antoine Pitrou @pitrou.net · 1d

Great news for Arrow (and more work for us 😄).
Also, those work items also implicitly apply to #ApacheParquet. @julien.ledem.net

QuantStack @quantstack.bsky.social · 1d

The Sovereign Tech Agency backs Apache Arrow's Future with QuantStack!

This funding will empower Arrow maintainers at QuantStack, @pitrou.net and @raulcumplido.bsky.social, to advance security, modularity, and long-term sustainability of @arrow.apache.org.

medium.com/@QuantStack/...

The Sovereign Tech Agency Invests in Apache Arrow’s Future with QuantStack

We are thrilled to announce that the Sovereign Tech Fund is providing critical funding to sustain and improve Apache Arrow, an essential…

medium.com

1 2 9

Reposted by Dewey Dunnington

Kyle Barron @kylebarron.dev · 1d

New blog post! kylebarron.dev/blog/fast-py...

This dives into performance pitfalls of Python bindings, why projects like Lonboard are so fast, and why I care about Apache Arrow so much. The post is high level and you don't need to know C/C++/Rust to understand it!

1 4 14

Dewey Dunnington @paleolimbot.bsky.social · 2d

FGB and COG are using the same mechanism that hyparquet, fsspec, objstore, and DuckDB's httpfs are using to read from http (i.e., range requests)

1

Dewey Dunnington @paleolimbot.bsky.social · 2d

Definitely depends what you're up to! People are definitely experimenting with serving features directly from Parquet files for visualization (e.g., medium.com/center-for-c... )

Optimal GeoParquet Partitioning Strategy

My quest for one-size-fits-all format

medium.com

1 2 2

Dewey Dunnington @paleolimbot.bsky.social · 3d

- FGB is great! Lookups with small ranges work well but full scans are much slower (particularly if zipped!)
- SedonaDB, DuckDB, and pyarrow with fsspec can all do partial Parquet reads from http (give geoarrow.org/data a try!)

Data – GeoArrow

geoarrow.org

1 1 1

Dewey Dunnington @paleolimbot.bsky.social · 4d

Interesting...I think of GeoParquet and GeoArrow as tech that allows more people to avoid the cloud (i.e., use your laptop to do things that everyone thought they needed "the cloud" for five years ago). When your 14 GB .fgb files become 2GB GeoParquets, you can fit a whole lot more of them locally!

mdsumner @mdsumner.bsky.social · 5d

"Do things like "GeoParquet" and "GeoArrow" really matter to practitioners who aren't working for a handful of hyper-scaling organizations?". 🤔 First I've heard my employer called that 😆

sgillies.net/2025/09/17/l...

Leaving the GDAL Project Steering Commitee

I've been on the Project Steering Committee (PSC) of the GDAL project for a few years now. It's a hugely significant software project, one that I'm very fond of, and I like to think that I've had an i

sgillies.net

1 4 11

Reposted by Dewey Dunnington

Kyle Walker @kylewalker.bsky.social · 5d

R-Universe by @rOpenSci is an extremely useful project.

My latest R package, {pmtiles}, is not a good fit for CRAN as it bundles a pre-compiled Go library.

With R-Universe, I was able to get installable cross-platform binaries built in hours!

#rstats

walkerke.r-universe....

2 11 40

Dewey Dunnington @paleolimbot.bsky.social · 9d

Last week Apache Sedona announced SedonaDB...and now I finally get to write about it! Here I use SedonaDB to demonstrate the explore/iterate/visualize cycle...SedonaDB's CRS support + @kylebarron.dev's `lonboard` is 👌 for exploring new data!)
dewey.dunnington.ca/post/2025/st...

Stream network traversal with SedonaDB | Dewey Dunnington

Last Wednesday the Apache Sedona project announced SedonaDB. There’s also a great post on the Whereobots blog that has a bit more context, or if you’re like me and you just want to see the code you ca...

dewey.dunnington.ca

2 9

Dewey Dunnington @paleolimbot.bsky.social · 13d

I cannot say enough about DataFusion...in order to build an engine that considers spatial types at every level we needed to customize types, functions, optimizer rules, joins, Parquet pruning, and more. DataFusion not only made this possible but documented even the most obscure bits. So cool!

Andrew Lamb @andrewlamb1111.bsky.social · 13d

"Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen"

Built in Rust with Apache DataFusion

sedona.apache.org/latest/blog/...

Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen - Apache Sedona

Apache Sedona is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark, Apache Flink, and Snowflake, with a set of...

sedona.apache.org

3 25

Reposted by Dewey Dunnington

Andrew Lamb @andrewlamb1111.bsky.social · 13d

"Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen"

Built in Rust with Apache DataFusion

sedona.apache.org/latest/blog/...

Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen - Apache Sedona

Apache Sedona is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark, Apache Flink, and Snowflake, with a set of...

sedona.apache.org

5 15

Dewey Dunnington @paleolimbot.bsky.social · 13d

I'm beyond excited for this! Built on Apache DataFusion and Apache Arrow, a composable single-node analytical database with spatial types as first-class citizens. GeoPandas IO, KNN joins, CRS support, R/Python bindings, and top-notch lazy GeoParquet reads.

sedona.apache.org/latest/blog/...

Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen - Apache Sedona

Apache Sedona is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark, Apache Flink, and Snowflake, with a set of...

sedona.apache.org

1 6

Reposted by Dewey Dunnington

Yutani @yutannihilation-en.bsky.social · 18d

100 stars🎉 Savvy is a niche framework for those who want to use Rust in an R package yet extendr is not their fit, so I didn't expect to see so much stars!

github.com/yutannihilat...

GitHub - yutannihilation/savvy: A simple R extension interface using Rust

A simple R extension interface using Rust. Contribute to yutannihilation/savvy development by creating an account on GitHub.

github.com

2 4

Reposted by Dewey Dunnington

Edzer Pebesma @edzer.mastodon.social.ap.brid.gy · 19d

Spatial Data Science across languages now has a website: https://spatial-data-science.github.io/ Thanks @martinfleis !

Spatial Data Science across Languages

spatial-data-science.github.io

8 6

Reposted by Dewey Dunnington

Bryce @brycemecum.com · Aug 28

Looking forward to attending this! #apachearrow

Apache Arrow @arrow.apache.org · Aug 28

Register for the Apache Arrow Summit Paris 25 (October 2nd) at: www.meetup.com/pydata-paris...
The event is hosted by @pydataparis.bsky.social
We are looking forward to seeing you there and talking about all things Arrow.

Apache Arrow Summit, Thu, Oct 2, 2025, 9:30 AM | Meetup

The day after the PyData Paris conference, we’re excited to host the first-ever Apache Arrow Summit - a gathering dedicated to fostering collaboration and innovation within

www.meetup.com

1 4

Reposted by Dewey Dunnington

Kyle Barron @kylebarron.dev · Aug 22

Published arro3-core v0.6 with wider dtype support for converting numpy ndarrays to Arrow. It now additionally supports:
- datetime64
- timedelta64
- fixed-width strings/bytes
- variable-width strings
- object arrays of str/bytes

And the Python wheel is still just 2.5MB!
github.com/kylebarron/a...

Release py-v0.6.0 · kylebarron/arro3

arro3-core Breaking Changes 🔧 No breaking changes. New features ✨ Expanded dtype support in Array.from_numpy. It now additionally supports datetime64, timedelta64, fixed-width strings and bytes, v...

github.com

1 2 9

Reposted by Dewey Dunnington

Columnar @columnar.tech · Aug 18

In September the @columnar.tech crew are headed to PyData Paris 2025 and the first ever Apache Arrow Summit. The organizer @quantstack.bsky.social is a dedicated supporter of @arrow.apache.org. We’re delighted to be sponsoring the event.

PyData Paris @pydataparis.bsky.social · Aug 18

Welcome to our new sponsor Columnar !

6 9

Reposted by Dewey Dunnington

Jakub Nowosad @jakubnowosad.com · Aug 6

Check out "Scaling the r-spatial ecosystem" by Dewey Dunnington 🌍📦
An exploration of how R’s spatial tools can be used for big(ger) data.

Video: youtu.be/tjNEoIYr_ag?...
Slides: dewey.dunnington.ca/slides/rspat...

#RStats #rspatial #GIS #SpatialData

1 12 16

Reposted by Dewey Dunnington

Yutani @yutannihilation-en.bsky.social · Jul 23

Glad to see another proof that Rust-powered R packages work fine on webR!

For those who are wondering what "savvy" is, it's a framework to use Rust in an R package. #rstats

yutannihilation.github.io/savvy/guide/...

1 2 6

Reposted by Dewey Dunnington

Etienne Bacher @etiennebacher.bsky.social · Jul 23

polars can be used via webR, thanks to the work of @gws.phd both on webR and on the Rust component of polars.

It also uses savvy, developed by @yutannihilation-en.bsky.social‬, to connect Rust and R.

2 6

Reposted by Dewey Dunnington

Etienne Bacher @etiennebacher.bsky.social · Jul 23

#rstats polars 1.0.0 is now available!

polars provides bindings to the eponymous Rust library, a very efficient dataframe library.

This is the outcome of a 1-year long rewrite led by @eitsupi to refactor the internals and match the API of Python polars.

pola-rs.github.io/r-polars/

Details ⬇️

Polars R Package

pola-rs.github.io

4 12 36

Dewey Dunnington @paleolimbot.bsky.social · Jul 22

A new version of @arrow.apache.org is out! I'm personally excited about the Parquet support for Geometry/Geography but there's a lot of other great features in this release...check out the release post for all the details! arrow.apache.org/blog/2025/07...

5 13

Reposted by Dewey Dunnington

Yutani @yutannihilation-en.bsky.social · Jul 22

Voting started! I hope my proposal of a talk about GeoArrow is attractive enough 🙏

FOSS4G 2025 Auckland @foss4g.org · Jul 22

Time to select what looks delectable to you 🧁

Wander through inspiring workshop, talk, & presentation submissions

Pause & vote up or vote down in ranking

Together = a collectively curated conference

Grab yourself a cool or hot drink and dive in:
https://2025.foss4g.org/attend/call-for-papers

3 4

Reposted by Dewey Dunnington

Apache Arrow @arrow.apache.org · Jul 19

Just a reminder that this call closes July 26! Get your proposal(s) in by then. If you have any questions or need help, feel free to DM.

Apache Arrow @arrow.apache.org · Jul 7

Apache Arrow Summit 25 is happening! Join us in person on October 2nd, in Paris (hosted by @pydataparis.bsky.social ). The Call For Proposal is open, submit your talks before July 26th:
sessionize.com/arrow-summit...

Arrow Summit 2025: Call for Speakers

sessionize.com

1 1

Reposted by Dewey Dunnington

Bryce @brycemecum.com · Jul 18

@duckdb.org just merged a PR for a new, simpler, and -- mostly importantly -- non-deprecated @arrow.apache.org C API: github.com/duckdb/duckd.... #apachearrow #duckdb

New Arrow C-API by pdet · Pull Request #18246 · duckdb/duckdb

This PR introduces the new Arrow C API, which is intended to replace the deprecated Arrow API. Why a new Arrow C-API? We decided to rewrite the Arrow C-API a while ago and marked all current method...

github.com

2 1 6