The argument is more Layers AND than Layers OR

We need to evolve data architectures.

So I wrote this: medium.com/thinking-abo...

I open up my browser and search for “data architecture” and back comes articles talking about Medallion Architecture, Source-Warehouse-Consumption Layers and so on.

I don’t see many alternatives.
Are we allowed to have a different opinion?

It probably started around 10 years ago when every Big Data/Hadoop project I was involved in seemed to mimic the good ol'data warehouse architectures of the 1990s. I was frustrated ...

Few typos above - apologies

I’m not saying that building a better LLM model is not important. Of course it is.

But for GenAI app developers it is less crucial.

Designing the app is paramount.

When I look at the features that OpenAI, Google and Anthropocene are launching they are more dependent on app features than LLM model

My argument was that I can build an app, swap out GPT4o for Claude 3.5 and get similar results.

You get better results from better prompting. Or getting another model to check your results.

- then I moved onto asking Gemini to read the code on my screen (in VSCode) and tell me what it did
- it did that pretty well (after an initial hallucination)

It's exciting!!!

- then I got it to explain the paper and research similar papers (ScholarGPT does this but not from a shared screen - you have to copy-paste text)
- and I asked it to identify gaps in the literature and research (it did a reasonable job)

I don't agree with everything Andrea Gioia writes but I do think this book is a valuable addition to the body of knowledge on data products, data management and data architectures.

Go read it!!

This is a lot of heavy stuff to get into with the first 3 chapters of a book.

I'll provide a word of warning as well - within the first 3 chapters we get a diagnosis of data platforms using System Dynamics, we get an interpretation of the organisation that builds them using Viable Systems Method and we get straight into Hexagonal Architecture as an approach to design.

Let me provide an excerpt from Chapter 1 to do with the failure of data platforms as monolithic architectures

There are too many books written that simply explain how to do things.

Now that's not a bad thing - the last book I mentioned, "Deciphering Data Architectures", is such a book. These books teach you stuff which is useful.

But "Managing Data as a Product" challenges you to re-think what you know.

Why should you read this?

If you liked Zhamak Dehghani's book on Data Mesh then this is (in my view) a book of the same ilk. It is not overtly data mesh, although it makes plenty of data mesh references.

"Of the same ilk" means that it challenges existing thinking on data and I like that.