Marcos Rubinelli
banner
marcosrubinelli.bsky.social
Marcos Rubinelli
@marcosrubinelli.bsky.social
Sr. Data Engineer @ RecargaPay
Likes Python, SQL, and dogs (not necessarily in that order.)
Incremental migrations with the strangler fig pattern are less risky than Big Bang migrations, but the problem I often see is that teams stop midway due to shifting priorities and end up with Rube Goldberg pipelines cobbled together with whatever the data team was using at the time.
November 24, 2024 at 8:47 PM
Fair enough. My experience is colored by my first contact with k8s, where an ingestion that could literally be a thirty-line script was turned into an entire cluster with its own queue and management database by a resume-driven engineer. But as a foundation for a sensible stack, k8s is fine.
November 24, 2024 at 2:28 AM
k8s is far from a dominant platform in data engineering. Most teams heavily rely on friendlier platforms like Databricks and Snowflake, for very good reasons. This may change when you have a large enough organization, but the average data team shouldn't waste time on sidecars and egress policies.
November 24, 2024 at 1:54 AM
You're spending weeks, maybe months of engineering time upfront for the dubious benefit of an easier cloud migration. You're also making expanding the team harder, because new team members have to know k8s on top of the usual stack. Embrace your cloud provider. You'll rewrite when you move anyway.
November 24, 2024 at 1:22 AM