Lightnews — Scholar-powered news

Andy Grove @andygrove.io · 3d

It’s steak night tonight and our dog is patiently waiting for her share.

1 11

Andy Grove @andygrove.io · 15d

I like the name “RAD stack” for this.

Yaroslav Tkachenko @sap1ens.com · 16d

Today, in the Data Streaming Journey, I'm sharing my experience of building data streaming products with the RAD stack: Rust, Arrow, DataFusion.

www.streamingdata.tech/p/streaming-...

Streaming and the RAD Stack

RAD: Rust, Arrow, DataFusion

www.streamingdata.tech

7

Andy Grove @andygrove.io · 21d

Check out the latest release of the Comet accelerator for Apache Spark

datafusion.apache.org/blog/2025/09...

Apache DataFusion Comet 0.10.0 Release - Apache DataFusion Blog

datafusion.apache.org

1 10

Reposted by Andy Grove

Yaroslav Tkachenko @sap1ens.com · 23d

Introducing Iron Vector: native, columnar, vectorized, high-performance accelerator for Apache Flink SQL and Table API built on top of Rust, Arrow and DataFusion.

Reduce your Flink compute cost by up to 2x or handle 2x more data with the same infrastructure.

1 5 17

Reposted by Andy Grove

Rust Language @rust-lang.org · 27d

We received reports of a phishing campaign targeting crates.io users. Do not click on links asking to authenticate to protect your account. More information: blog.rust-lang.org/2025/09/12/c...

crates.io phishing campaign | Rust Blog

Empowering everyone to build reliable and efficient software.

blog.rust-lang.org

58 110

Reposted by Andy Grove

Andrew Lamb @andrewlamb1111.bsky.social · Sep 4

Thanks to @clflushopt.bsky.social, make massive TPCH datasets with tpchgen-cli 2.0:

SF1000 (1TB raw, 220GB in @ApacheParquet ) in less than 10 mins (6m45s) on aging laptop

Try it now:

pip install tpchgen-cli
tpchgen-cli --scale-factor 1000 --parts 100 --format=parquet

github.com/clflushopt/t...

1 4

Reposted by Andy Grove

Phil Eaton @eatonphil.bsky.social · Sep 4

I've been helping our analytics team integrate our DataFusion-based query engine for Postgres into EDB Postgres Distributed and finally here's an end-to-end demo.

You get HA Postgres plus seamless replication and DataFusion-based queries. This query turned out 6x faster than PG.

1 4 14

Andy Grove @andygrove.io · Aug 22

How my day is going

6

Andy Grove @andygrove.io · Aug 20

We now have a roadmap section in the Comet contributor guide, in case anyone was wondering what we are focusing on lately and what features will be arriving in future releases.

datafusion.apache.org/comet/contri...

Comet Roadmap — Apache DataFusion Comet documentation

datafusion.apache.org

6

Reposted by Andy Grove

Alex P @ifesdjeen.bsky.social · Jul 18

Cassandra Team at Apple is searching for a fresh grad / person early in their career to join our ranks in SF/Bay Area!

Come work on super interesting problems with world class team. Help us build better Cassandra!

Ping me if you’re interested!

jobs.apple.com/en-us/detail...

Software Engineer, ASE Cassandra Storage - Jobs - Careers at Apple

Apply for a Software Engineer, ASE Cassandra Storage job at Apple. Read about the role and find out if it’s right for you.

jobs.apple.com

10 15

Andy Grove @andygrove.io · May 2

It took me a really long time to understand the flow of execution between JVM and native code during query execution in Comet. I wish I had thought about adding a tracing capability earlier.

github.com/apache/dataf...

perf: Add performance tracing capability by andygrove · Pull Request #1706 · apache/datafusion-comet

Which issue does this PR close? Closes #1705 Rationale for this change This feature makes it possible to visualize the flow of calls during query execution. What changes are included in this PR?...

github.com

1 6

Reposted by Andy Grove

Tim Saucer @timsaucer.bsky.social · Apr 7

We're pleased to announce that Apache DataFusion in Python 46.0.0 is released! Since the last announcement post we've had a lot of great features and new contributors. Please check out the blog post with details.

datafusion.apache.org/blog/2025/03...

#DataFusion #Python #DataFrame #PyData #Apache

Apache DataFusion Python 46.0.0 Released - Apache DataFusion Blog

datafusion.apache.org

2 5

Andy Grove @andygrove.io · Apr 2

We have a position open in the Spark team at Apple, in our Cupertino, CA office. The role would include working on Apache DataFusion Comet.

jobs.apple.com/en-us/detail...

Senior Software Development Engineer (Apache Spark) - Apple Data Platform - Jobs - Careers at Apple

Apply for a Senior Software Development Engineer (Apache Spark) - Apple Data Platform job at Apple. Read about the role and find out if it’s right for you.

jobs.apple.com

6 15

Andy Grove @andygrove.io · Mar 21

We have TPC-H benchmarks for single node with a small scale factor in the contributors guide. We only benchmark against Spark though and not against Spark RAPIDS.

datafusion.apache.org/comet/contri...

Apache DataFusion Comet: Benchmarks Derived From TPC-H — Apache DataFusion Comet documentation

datafusion.apache.org

1

Andy Grove @andygrove.io · Mar 21

Here's the blog post announcing Comet 0.7.0

datafusion.apache.org/blog/2025/03...

1 7

Andy Grove @andygrove.io · Mar 19

I hate to say it, but "it depends". I'd recommend running your own benchmarks for your specific workloads. Performance will also vary greatly by environment (number of CPUs vs GPUs, different GPU types, and so on).

2

Andy Grove @andygrove.io · Mar 19

DataFusion Comet 0.7.0 is now available in Maven. We'll be publishing a blog post next week with all the details.

The repo has been updated with the latest benchmark results. For single executor TPC-H @ 100 GB, we now see a 2.2x increase over Spark (up from 2x in 0.6.0).

github.com/apache/dataf...

GitHub - apache/datafusion-comet: Apache DataFusion Comet Spark Accelerator

Apache DataFusion Comet Spark Accelerator. Contribute to apache/datafusion-comet development by creating an account on GitHub.

github.com

1 1 12

Andy Grove @andygrove.io · Feb 18

One month on, and I have zero regrets about quitting Facebook & Instagram.

I have replaced the scrolling time with listening to podcasts.

I now stay in touch with family overseas via email and photo sharing, and I use Snapchat for sharing photos with immediate family, privately. Works great.

Andy Grove @andygrove.io · Jan 18

I've finally decided to quit using Facebook. My feed is overwhelmed with nonsense content that I am not interested in and cannot seem to block.

It is a real shame, though, because it was a good way to stay connected with family.

Is there a viable alternative? What are others using instead?

1 15

Reposted by Andy Grove

Commonhaus Foundation @commonhaus.org · Feb 18

Chris Riccomini (@chris.blue) shares his thoughts on Open Source foundations: Apache, CNCF, Commonhaus. He also explains why Commonhaus is a better fit for SlateDB

cnr.sh/posts/compar...

Comparing Apache, CNCF, and Commonhaus | cnr.sh

I've used open source projects for over 30 years and contributed for about 20 of those. My first interaction with an open source foundation was with Apache when I began working with Apache Hadoop ...

cnr.sh

6 14

Andy Grove @andygrove.io · Feb 18

Comet 0.6.0 has been released. This is a smaller release than usual now that we have moved to an approximately monthly release cadence to match core DataFusion.

datafusion.apache.org/blog/2025/02...

Apache DataFusion Comet 0.6.0 Release - Apache DataFusion Blog

datafusion.apache.org

6

Andy Grove @andygrove.io · Feb 12

Ballista 43.0.0 has been released, and now provides seamless integration with DataFusion.

datafusion.apache.org/blog/2025/02...

Apache DataFusion Ballista 43.0.0 Released - Apache DataFusion Blog

datafusion.apache.org

16

Andy Grove @andygrove.io · Jan 29

Check out this excellent presentation from @robtandy.bsky.social on his work with the DataFusion Ray project from last week's DataFusion community meetup.

It is a great overview of how to build a distributed system on top of DataFusion.

www.youtube.com/watch?v=ceTo...

Apache DataFusion Community Meeting 2025/01/22 08:57 MST - Recording

YouTube video by Datadog

www.youtube.com

1 2 11

Andy Grove @andygrove.io · Jan 26

This Week in DataFusion Comet (Jan 26):

github.com/apache/dataf...

This Week in Comet (Jan 26) · Issue #1342 · apache/datafusion-comet

Introduction These notes reflect things I am personally involved in or thinking about and may not cover all activities. Feel free to add comments for anything that I missed. Previous week's issue: ...

github.com

6

Andy Grove @andygrove.io · Jan 23

Is this using Arrow and/or DataFusion? If so, our Discord is probably a good place to ask.

datafusion.apache.org/contributor-...

Communication — Apache DataFusion documentation

datafusion.apache.org

1

Andy Grove @andygrove.io · Jan 18

I've finally decided to quit using Facebook. My feed is overwhelmed with nonsense content that I am not interested in and cannot seem to block.

It is a real shame, though, because it was a good way to stay connected with family.

Is there a viable alternative? What are others using instead?

4 6