Author | Lightnews

Ananth Packkildurai @ananthdurai.bsky.social · 6d

This is the most personal essay that I have written in Data Engineering Weekly. I shared a few key moments in my life and how fortunate I was to meet mentors along my professional journey, which shaped my career.

Thinking Like a Data Engineer

A Journey Beyond Code — Toward Systems, Curiosity, and Confidence

www.dataengineeringweekly.com

9

Ananth Packkildurai @ananthdurai.bsky.social · 11d

🚀 Data Vault vs. Dimensional Modeling vs. Medallion Architecture — When viewed through a modern enterprise data lens, these techniques interlock.

I break down how in Part 2 of my “Revisiting the Medallion Architecture” series.

Revisiting Medallion Architecture: Data Vault in Silver, Dimensional Modeling in Gold

How to Balance Flexibility and Performance in a Modern Data Platform

www.dataengineeringweekly.com

4

Ananth Packkildurai @ananthdurai.bsky.social · 12d

Fivetran and dbt form a strong foundation for modern data infrastructure, known for bringing simplicity to complex engineering workflows. That said, calling it “open” data infrastructure feels like a stretch.

3 5

Ananth Packkildurai @ananthdurai.bsky.social · 15d

Should we update the definition of an "Analytical Engineer"?

4

Ananth Packkildurai @ananthdurai.bsky.social · 20d

As a data engineer, you can't treat zero-party (consent) and third-party (inferred) data the same way. This distinction is critical for building systems that are scalable, private, and trustworthy.

Here’s my guide:

Engineering Growth: The Data Layers Powering Modern GTM

Building privacy-preserving pipelines that unify zero-, first-, second-, third-, and fourth-party data into a coherent GTM ecosystem.

www.dataengineeringweekly.com

5

Ananth Packkildurai @ananthdurai.bsky.social · 24d

Could be. Composable CDP has not gained significant market share, as identity resolution is a key component that is often proprietary.

1

Ananth Packkildurai @ananthdurai.bsky.social · 25d

With Census already in with Fiveatran and with dbt, it is most likely to evolve as a composable CDP.

1 1

Ananth Packkildurai @ananthdurai.bsky.social · 26d

Airbnb: Real-Time Key-Value Store

Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.

Read the full story here: www.dataengineeringw...

1

Ananth Packkildurai @ananthdurai.bsky.social · 27d

Read the full story here:

Data Engineering Weekly #239

The Weekly Data Engineering Newsletter

www.dataengineeringweekly.com

Ananth Packkildurai @ananthdurai.bsky.social · 27d

Grab: Partner Gateway Metrics at Sub-Second Speed
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.

1

Ananth Packkildurai @ananthdurai.bsky.social · 29d

💡 Read the full story →

Data Engineering Weekly #239

The Weekly Data Engineering Newsletter

www.dataengineeringweekly.com

1

Ananth Packkildurai @ananthdurai.bsky.social · 29d

Netflix Muse: Scaling Analytics at Trillion-Row Scale
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.

1

Ananth Packkildurai @ananthdurai.bsky.social · Sep 29

🔗 Link in bio:

Data Engineering Weekly #239

The Weekly Data Engineering Newsletter

www.dataengineeringweekly.com

Ananth Packkildurai @ananthdurai.bsky.social · Sep 29

⚡ Latency Every Data Streaming Engineer Should Know

“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!

1

Reposted by Ananth Packkildurai

Chris @chris.blue · Sep 27

I enjoyed this post by @ananthdurai.bsky.social. Does a great job tying a bunch of recent papers and concepts together.

What “Supporting Our AI Overlords” and “Semantic Spacetime” Tell Us About the Future of Data Infrastructure

Connecting agent-first and universal semantic grammar to reimagine data infrastructure beyond the relational model.

www.dataengineeringweekly.com

1 8

Ananth Packkildurai @ananthdurai.bsky.social · Sep 27

🔗 Full story:

This MCP Server Could Have Been a JSON File

There's a lot of buzz around MCP. I'm not convinced it needs to exist.

materializedview.io

2

Ananth Packkildurai @ananthdurai.bsky.social · Sep 27

MCP (Model Context Protocol) promises a new way for LLMs to use tools.

Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs

So… could MCP have just been a JSON file?

💡 More insights: www.dataengineeringw...

1 2 3

Ananth Packkildurai @ananthdurai.bsky.social · Sep 26

How Tables Got Smarter: Iceberg → DuckLake. From static snapshots to stream-native updates and catalog-first metadata, tables are evolving fast. Choose by intent, not hype.

Subscribe → www.dataengineeringw...

Full story → medium.com/fresha-da...

3

Ananth Packkildurai @ananthdurai.bsky.social · Sep 25

I wrote my thoughts on Supporting Our AI Overlords.

What “Supporting Our AI Overlords” and “Semantic Spacetime” Tell Us About the Future of Data Infrastructure

Connecting agent-first and universal semantic grammar to reimagine data infrastructure beyond the relational model.

www.dataengineeringweekly.com

4

Ananth Packkildurai @ananthdurai.bsky.social · Sep 25

How Tables Grew a Brain: Iceberg → DuckLake
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.

More insights → www.dataengineeringw...

Full story → medium.com/fresha-da...

2

Ananth Packkildurai @ananthdurai.bsky.social · Sep 24

BlaBlaCar scales like a pro!

dbt Core → Transform like a champ

Airflow → Orchestrate effortlessly

CI/CD → Deploy instantly

Dev Containers → Standardized dev

📖 Full story →medium.com/blablacar...

💡 More insights → Subscribe to DEW

#DataEngineering #dbt #Airflow #CICD #DevContainers

1

Ananth Packkildurai @ananthdurai.bsky.social · Sep 23

🚀 AI adoption is booming—but most data isn’t ready!

AI-ready data is:

Unified

Real-time

Human-verified

Governed

Without it, AI can confidently fail. With it? Reliable, scalable results.

📖 Read More

💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering

Ananth Packkildurai @ananthdurai.bsky.social · Sep 22

💡 More insights → Data Engineering Weekly
💡 Learn more: stripe.com/blog/how-...

#DataEngineering #Stripe #RealTimeAnalytics #ApacheFlink

How we built it: Real-time analytics for Stripe Billing

Among global business leaders surveyed, 84% agree that adapting pricing quickly will be a key competitive advantage. Our new real-time analytics system for Stripe Billing helps them spot customer trends just as they emerge.

stripe.com

Ananth Packkildurai @ananthdurai.bsky.social · Sep 22

Stripe’s Real-Time Billing Analytics ⚡
Content:
Stripe wanted real-time visibility into subscriptions.
Traditional batch systems weren’t fast enough. ⏱️
They built a pipeline using Flink, Spark, and Pinot v2.
Now, analytics arrive in minutes, not hours. Queries return in <300ms. 🚀

1 1 2

Ananth Packkildurai @ananthdurai.bsky.social · Sep 22

The 238th edition of Data Engineering Weekly is available, featuring exciting Data & AI articles.

Read more:
www.dataengineeringw...

1 5