after a few failed attempts at tone editing, this was unfortunately effective

Real talk: when is try/except to suppress an exception lazy (compared to checking the upstream data) vs the most efficient implementation?

I've done this plenty of times to only call out an API once instead of twice, but now looking back it feels like bad practice

#dataBS

Yeah I used (or tried to use) Airflow but it kept eating so much RAM I could not do anything else on my laptop
Switched to prefect and never looked back
I wish it was more widespread in the industry

we live in the land where bon jovi and pitbull can do a collab for a new rendition of now or never. the american empire not only will last 1000 years, but it deserves to.

Alright #dataBS: for anyone using Databricks -

What do you mostly use it for? What made you choose the tool? Where do you find it solving your problems most?

I feel like 2022 was the year we tried to solve lineage with observability tools, got decently far but not far enough to fully understand failures, so we settled for alerting on failures instead.

Is 2025 going to be the year we solve for true lineage outside of the data warehouse?

#dataBS

Going from dev to prod in literally anything should be easy.

This is still an unsolved problem.

Devops is a blocke, and data teams are still trying to figure out IaC.

On my mind today: something as sinple as dynamic work pools in Prefect could solve this.

#dataBS

Auto spin up / spin down is not flashy these days, it's table stakes. Infra is so expensive - anything that can help save infra cost pays dividends.

Enjoyed this piece by @sarahkb.bsky.social on measuring PLS vs PLG, where the later mostly doesn't work for enterprise sales. We had to figure out how to measure PLS early because Coginiti targeted customers in highly secure industries like gov, finance, & insurance.

Event based != realtime

Event based jobs are often batch, with event triggers used to optimize running only when needed (most common).

True realtime reqs are concentrated more in certain industries - finance, logistics, user-facing analytics (and I'm sure others I missed).

#dataBS change my mind

Alright #databs time for a trial run

What one type of technology would you consider the heart and soul of your data stack?

So I guess this is now the place I get to die on my small hills

Let's start with a banger

Hosting open source software is 9 times out of 10 not a value add for your data team