Modern times, old vibes. Image generated with gpt4o
Something that intrigues me deeply is our knack for latching onto the next fashionable topic while stubbornly clinging to yesterday’s mindset. Perhaps it’s just a matter of time—our values and habits evolve more slowly than the industry around us.
A few years ago the term “Modern Data Platform” (MDS) became popular. It proposed a layered technology + process architecture that decoupled transactional databases from analytical environments. At the center sat a data lake, where raw information lived in object storage (e.g., S3, Cloud Storage, or NFS). Instead of the classic extract–transform–load (ETL) pipeline, data would now be extracted, loaded, and only then transformed—a sort of just-in-time transformation for analytics.
The catch was that this setup still needed a final consumption layer, usually filled by a column-oriented database. That technological fragmentation has since turned into the Lakehouse paradigm championed by Databricks, unifying the data-warehouse engine and advanced analytics under one roof.
In other words, the entire spectrum—from ingestion all the way to exploitation—now lives inside a single technology block. Data can exist in multiple versions (from raw to highly curated). The latter are stored in a columnar format on top of cheap, scalable storage, with niceties such as partitioning and ordering. This turns a plain data dump into something closer to a database, achieving several goals at once:
-
massive semistructured datasets stay intact,
-
storage and compute remain separate (unlike in a traditional columnar DB), and
-
data transfer and storage stay efficient.
Several open table formats have gained traction— Hudi, Delta, Paimon o Iceberg —but Iceberg has clearly become the de-facto standard. Even vendors, like AWS with S3 Tables, offer abstraction layers over them solving some operational challenges (A short overview on Lakehouse open formats).
Technology ≠ Success
So, the tech problem seems “solved”: a more modular stack, open standards, easier interoperability, and lower costs. Yet the data-lake model relies on decentralizing responsibility for both ingestion and consumption—and that’s where the trouble starts. Conceptual frameworks such as Data Mesh address this beautifully on paper, but the devil lives in the real-world details. Ultimately we need the domains that produce and consume data to act as autonomously as possible, shattering the old setup where a central Business Intelligence team owned extraction and analytics.
To keep these initiatives from failing, we need organizational and cultural shifts affecting not just tech (operations and engineering) but also analytics functions (data science and business analysis). Key change vectors include:
- Ownership. Data producers must feel accountable for the data products they publish.
- Catalog & governance. Proper metadata, quality checks, and clear ownership lines are non-negotiable.
- Enablement teams. Platform squads should provide self-service tools so non-specialist teams can federate their data.
- Upskilling. Broader education in data engineering and data exploitation.
Success is measured by the adoption
A technology piece is not good or bad, it just serves a purpose. If what’s on the table is not valid you have the wrong stack, no matter how modern it is.
Centralized or Decentralized? that’s the question.
Well, there’s no one-size-fits-all answer. Less mature or smaller organizations may start with a more centralized model; larger firms might adopt a hybrid; the most mature can fully federate. To me, what is essential is strong leadership for the company’s data function—not to centralize ownership, but to steer strategy and coach teams as they adopt the new operating model. The decision driver of decentralization should the the degree of complexity in the various axes involved data: organizational, structural or procedural. When complexity increases the more need for a The Composable Data Platform.
Conclusion
It’s hard to draw a clear conclusion, given the paradigm is still evolving. However, what we can say is the Modern Data Platform provides an nice technological framework that overcomes the traditional limitations of data-warehousing and advanced analytics. But it’s no silver bullet. Real effectiveness comes only when the right organizational changes take root—and, above all, when we genuinely embrace a new way of working, complete with fresh responsibilities and the powerful tools now at our disposal.