Following our discussions about data identification and classification and data privacy issues in the previous sessions of our ‘Operationalising Data Governance’ webinar series, the focus of the fourth session was on technical data lineage.
Joined by panellists Jan Ulrych, VP of Research and Education at Manta and Markus Buhmann, Data Leader and Recoving Architect, we explored data lineage, its benefits, challenges, and its crucial role in regulatory compliance and internal efficiency.
The emergence and importance of technical data lineage
The webinar kicked off with a focus on the growing significance of technical data lineage in today’s digital age. The panellists and I discussed how this concept is gradually being recognised as a vital component of effective data governance. Jan Ulrych defined technical data lineage as a detailed map that illustrates the movement and transformation of data within an organisation. He stressed that it provides unprecedented visibility into data environments, thereby playing a crucial role in supporting other governance projects.
Markus Buhmann echoed these sentiments, adding that while lineage mapping might provide a sense of security, its real value lies in how it underpins other governance projects. He emphasised that lineage mapping often acts as a supporting process rather than a budgeted project, highlighting its integral role in efficient data management.
The benefits and challenges of lineage mapping
As the discussion progressed, we delved deeper into the potential benefits of lineage mapping. I highlighted how this tool could be utilised in change management to identify unused or unnecessary data assets, leading to significant cost reductions. Markus agreed, adding that understanding data flows during migration or transformation projects can increase efficiency and facilitate value-driven decision-making.
Jan identified key use cases of lineage mapping, including backing governance initiatives and boosting internal efficiency. He noted that understanding who uses the data and for what purpose is crucial for prioritising repairs or investigations. However, Markus raised a point of caution about the challenge of information overload when attempting enterprise-wide mapping without specific, defined needs.
Implementing data lineage: Complexities and considerations
The conversation then veered towards the complexities associated with implementing data lineage. Markus pointed out the challenges stemming from the abundance of applications used in organisations, particularly in scraping and dumping metadata into other tools. He also highlighted the difficulties in keeping track of system changes and visualising the movement of data.
Jan, on the other hand, stressed that some organisations might not derive the expected value from data lineage due to a lack of clear goals and improper scaling approach. He underscored the need to define clear quantification of value derived from governance to spur adoption.
The role of data mesh, federation concepts, and metadata-driven approaches
Both Jan and Markus agreed on the importance of data mesh and federation concepts in enhancing data governance. They emphasised the role of domain-driven design and ubiquitous language in operationalising good data governance practices.
I reiterated the need for consistent guidelines across the board in a federated data approach. I also hinted at possible setbacks of data mesh arising from varying toolsets used by different units of the organisation.
The discussion then shifted towards the importance of a metadata-driven approach in collecting data close to real time and maintaining data quality. Markus emphasised the importance of foundational data quality and the approach being incremental, and value-driven.
Navigating challenges and emphasising automation
Addressing the challenges of data silos and legacy systems was another important aspect of our discussion. Markus shared his experiences and recommended careful and precise steps, tailoring the approach based on specific needs and goals. Jan supported this view, highlighting the need for automation and choosing the right toolsets to manage the level of detail required for lineage.
However, both panellists agreed that achieving technical lineage is an ongoing process due to the constantly changing and evolving data environments. They emphasised the need for continuous monitoring and updates to ensure the accuracy and usefulness of the lineage maps.
Thus, the penultimate session of the series served as a comprehensive exploration of technical data lineage. It reinforced the importance of data lineage, and gave insights into the benefits of lineage mapping, implementation of data lineage and overcoming the challenges associated with it.
Looking for a detailed breakdown of this topic?
You can watch the webinar here:
Operationalising data governance can be overwhelming, but it doesn’t have to be. Our team of experts stand ready to help:
