opinion

Nov 3, 20256 mins

When tech giants likeNetflix and Intuit adopted the data mesh architecture, it looked like the next big thing. A few short years later, disillusionment has set in, with companies turning their backs on data mesh and industry experts writing it off as a fad or declaring it dead.

Amidst all of these conversations, if you are still thinking whether you should adopt the data mesh architecture and wondering if it will survive the test of time, then this post is for you.

How data mesh gained popularity

Let’s rewind a bit and see how data mesh architecture arose in the first place. In the late 2010s, many companies were doubling down on migrating data to a data lake and building workflows. Data lake architecture promised a perfect solution for companies wanting a centralized repository for all data to be used in analysis. But by the early 2020s, the limitations of the data lake system became clear, and loopholes in the approach came to the forefront.

One primary loophole was that the data lake was built and maintained by a separate engineering or analytics team, which didn’t understand the data in depth as thoroughly as the source teams. Typically, there were multiple copies or slightly modified versions of the same data floating around, along with accuracy and completeness issues. Every mistake in the data would need multiple discussions and eventually lead back to the source team to fix the problem. Any new column added to the source tables would require tweaks in the workflows of multiple teams before the data finally reached the analytics teams. These gaps between source and analytics teams led to implementation delays and even data loss. Teams began having reservations about putting their data in a centralized data lake.

Data mesh architecture promised to solve these problems. A polar opposite approach from a data lake, a data mesh gives the source team ownership of the data and the responsibility to distribute the dataset. Other teams access the data from the source system directly, rather than from a centralized data lake. The data mesh was designed to be everything that the data lake system wasn’t. No separate workflows for migration. Fewer data sanity checks. Higher accuracy, less duplication of data, and faster turnaround time on data issues. Above all, because each dataset is maintained by the team that knows it best, the consumers of the data could be much more confident in its quality.

Why users lost faith in data mesh

But the excitement around data mesh didn’t last. Many users became frustrated. Beneath the surface, almost every bottleneck between data providers and data consumers became an implementation challenge. The thing is, the data mesh approach isn’t a once-and-done change, but a long-term commitment to prepare a data schema in a certain way. Although every source team owns their dataset, they must maintain a schema that allows downstream systems to read the data, rather than replicating it. However, a general lack of training and leadership buy-in led to improper schema planning, which in turn led to multiple teams performing similar actions on the same data, resulting in duplication of data and effort and increased compute costs.

A lack of coordination across teams often led to incomplete and disconnected tables. Analytics tables that were missing columns or couldn’t be connected to each other were a disaster, putting everyone back at square one. Incomplete tables triggered the same old pattern that afflicted the data lakes, where every team began building its own layer on top of the source systems and populating required columns. Once again, teams began duplicating the datasets, defeating the whole purpose of the architecture.****

So was data mesh just a fad?

No, data mesh is not a fad, nor is it the next big thing that will solve all of your data challenges. But data mesh can dramatically reduce data management overhead, and at the same time improve data quality, for many companies.

In essence, data mesh is a shift in mindset, one that completely changes the way you view data. Teams must envision data as a product, continuously showing commitment for the source team to own the data set and discouraging duplication. When developing a new feature, teams must treat analytics data requirements as first-class citizens. That is, they must design the data schemasuch that all downstream requirements are met.

For example, if you convert your manufacturing datasets into a data mesh architecture, then your purchase order table must have all the columns required by finance, marketing, procurement, shipping, and analytics. No team should require further copying of this table or have to do any transformations on top of it to make the table usable for them.

A data mesh is not the right approach for everyone. For small teams with a limited number of datasets, it still makes sense to create a centralized data lake rather than having multiple workflows for every source team. But for large enterprises with huge datasets, where multiple teams make changes to the same source datasets regularly, decentralization can be extremely effective. It makes sense for the source team to build a complete dataset itself, rather than have every team copying the table, often doing transformations on top of it. In addition to wasting compute resources, this often introduces errors in the final dataset.

Data mesh architecture reduces the number of hoops required to access data and increases data accuracy. Companies that have done data mesh right have seen excellent results. For instance, a leading bank implemented a data mesh and saw a 45% reduction in the time taken to complete operational activities. If your company has the right use case and the right mindset, a data mesh could unlock easier access to higher-quality data for your analytics teams, allowing them to achieve better results with far less effort.

by Nidhin Ponon

Contributor

Nidhin Ponon has extensive IT experience of 20+ years across various domains and technologies, with data analytics and AI being his core areas of expertise. He has contributed to major data engineering and AI projects for tech giants including Meta, WarnerMedia, and Honeywell.