[Recipe] Lineage granularity

Modified on: Tue, 30 Nov, 2021 at 1:59 PM

Why is this important?

Lineage is a powerful tool for understanding and ownership, for both functional and technical users.

However, when we start to establish the lineage of objects, we are faced with the following question: what granularity should we achieve?

The answer to this question is very important. It depends very strongly on:

The selected use case, in particular the target audience
Taking into account constraints: maturity of the organization, maintenance capacities ...

The ability to deliver adequate lineage is however essential for many projects. It allows, in particular, to:

Lineage construction is therefore a function of several factors. Here are a few:

The selected use case: undoubtedly the most important. A more BI-oriented project will need a high level of granularity, especially on technical objects, which is not necessarily the case for a governance project which will instead focus on functional modules.
The maturity of the organization: the more mature the organization is in terms of data management, finer will be the granularity.
Maintenance capabilities: Very fine granularity implies high construction and maintenance costs. This question is also to be put in relation with maturity: a significant maturity generally allows to have strong maintenance capacities.

Let us take two extremes as an example:

A complete lineage including detailed field-to-field links with details of the processing units.

These two lineages naturally meet different needs.

	Complete lineage	Simple lineage
Need	Need for details to document / collaborate on work in progress. It allows to To integrate newcomers more easily Identify management responsibilities Functionally model my objects to better discuss with functional interlocutors	Need for awareness / acculturation around data. It allows you to answer questions such as: What is the meaning of my data? What is their origin? Who to ask? Which reports can be of interest?
Target audience	Technical profile	Business professional profile
Positive points	In-depth understanding of data flows and transformations Can serve as a basis for collaboration between technical users	Quick ownership of the subject Limited risk of error Simple construction, low maintenance
Negative points	Complex construction Maintenance costs Complex to understand for functional users	Low added value for technical users The representation loses value over time and along with mapped objects: everything points to the same level

The reflection on the granularity of the lineage can be deepened with the notion of workspace:

Workspaces with functional oriented lineages, allowing a better appropriation of the subject
More complete spaces for technical users