As our innovative Data Governance-as-a-Service (DGaaS) managed service offering gains a wider appeal in today’s business context, one of the hottest discussions that I have been involved with both internally and externally is around data mesh and data fabric. What are they? Can we buy one? How do we implement them? What is best for us?
Having read numerous blogs, papers and listened to some key notes by vendors, integrators and consultants along with playing back customer conversations, I have (hopefully) simplified what was becoming a recurring headache for myself, as well as those who I have been interacting with.
What is Data Fabric and Data Mesh?
They are both data architecture concepts for building scalable, accessible, better managed and governed data environments. While they share similar objectives, they differ significantly in their approach.
A data fabric is a technology-driven approach for unifying metadata from multiple tools/systems to provide a single view of the truth. It aims to provide a single, virtual layer on top of distributed data where users access a single source of truth that is generally managed top-down within an organisation. The key benefits are unified visibility and timely access to data assets.
On the other hand, a data mesh is a people-driven approach focused on decentralising control, empowering teams to have ownership over their domain-specific data. The aim is to treat data as a product under the ownership of one team/group who act as the domain-specific data specialists to curate, develop and govern their controlled data set appropriately. Essentially, data mesh empowers domain-specific groups to manage their own data, treating it as a product, as they know their domain the best.
Put simply, a data fabric aims to provide a single, virtual layer on top of distributed data to ease management, whereas a data mesh actively encourages domain specific groups/ teams to manage data as they deem best for their use case albeit with some common overall governance.
Why Are They Important?
Data is the lifeblood of any organisation. However, with the exponential growth rate of data, the need to access it quickly and efficiently has become more challenging than ever before.
To address these challenges, both models offer scalable solutions that can effectively manage and govern data workflows. By unifying metadata or decentralising data control, these models can provide a single source of truth for data assets which can improve decision-making and collaboration across an organisation. Ultimately, implementing a data fabric or data mesh can help an organisation better leverage their data assets to drive business outcomes and achieve strategic objectives.
Can You Purchase Data Fabric and Data Mesh?
Quite simply – no you cannot!
No single vendor can sell you a data fabric or data mesh as they are architectural concepts which cannot be purchased as off-the-shelf products. However, whichever approach you choose, you will require tooling to help manage and automate your data estate (which can be bought), and business processes that need to be established.
How to Determine the Best Approach for Your Organisation?
Unfortunately, there isn’t a simple answer as both data mesh and a data fabric are concepts that aim to address the challenges of managing and accessing data in today’s complex data environments. While data fabric focuses on providing a unified view of data through metadata unification and centralised control, data mesh empowers domain-specific teams to take ownership of their data and manage it as a product.
The key to understanding what way to go, and which architectural frameworks and technologies to deploy is by finding out the key drivers for data use within, and what works best within your specific organisation.
The following questions may help determine the best approach for your organisation:
- Does my organisation operate as a decentralised structure?
- Does it have mature data governance processes?
- Is my data team overwhelmed with requests to distribute data?
- Do we struggle with poor data quality challenges and feedback when we produce and distribute data?
However, at a certain point of data maturity, a data fabric will become relevant for every organisation as no single tool can provide a single view of the information required by an organisation. Similarly, a data mesh may become crucial to achieving quality data and fulfilling data requests in a timely manner. In my opinion, they both have a place within organisations and are complimentary to each other.
Overall, understanding the differences between data mesh and data fabric, and asking the right questions to determine which approach is best for your organisation, can help you create a scalable, accessible, governed and secure data environment that meets your business needs.
Curious to learn more or need assistance operationalising this within your organisation? We can help you get started:
