Metaplane’s New Column-Level Lineage Map

If you’re a modern company, your data probably flows in complex ways through your data stack. Tracing that data, from tables and views all the way to BI tools, can feel like a scavenger hunt—and not always the fun kind.

One of the biggest benefits—and the biggest drawbacks—of the modern data stack is the proliferation of best-in-class tools to get the job done. One (or several) warehouses store and process the data. Other tools transform or move that data from place to place, and some, like Metaplane, help ensure that the data is current, correct, and complete. At the other end of the stack, stakeholders consume the data through one (or multiple) different BI tools.

It is, undoubtedly, a great benefit to have access to so many tools that each do their jobs so well. But, on the other hand, those myriad tools can make it quite a chore to get a holistic view of how your data flows. And if you want to

1. prevent downstream issues,
2. diagnose a root cause, or
3. ensure heavily-referenced tables, views, and columns are monitored in Metaplane
4. raise awareness about data at your company

…having a holistic view of your data is critical.

That’s why we’re thrilled to unveil Metaplane’s new lineage map, which lets your team explore your lineage from the source all the way to the BI tool.

We are even more excited that our lineage visualization makes common tasks, such as switching from table-level lineage to column-level lineage, simple and intuitive.

Prevent downstream issues

Let’s say your team wants to delete a column in one of your tables. To the uninitiated, that might seem like the sort of thing that should be pretty simple. But if you’re a seasoned data engineer, you know that when that column is referenced in dozens (or hundreds) of downstream dashboards, finding and ensuring that all of those references are updated can be quite the feat.

Now, using Metaplane’s lineage tool, you can see everything that depends on a particular table, view, or column. And to help make your exploration more fruitful, we’ve built in useful controls that allow you to filter by the number of levels and the object types you wish to see.

This helps take the guesswork out of many common tasks, such as refactoring dbt models, migrating data, and renaming or removing objects within your data ecosystem.

This is equally true in the case of a data incident. When your team has an active incident in Metaplane, Metaplane will highlight the downstream objects that are affected by the anomalous data. This makes understanding what’s affected and alerting the appropriate stakeholders a breeze, leaving you more time to triage and fix the issue.

Diagnose the root cause

When a data anomaly occurs, being able to quickly identify the root cause of an issue makes finding a solution infinitely easier. That’s why we highlight all affected items during a data incident, which helps you home in on exactly what upstream data might be experiencing the same anomaly.

So, for example, if both your Revenue and Weekly_Sales tables are both experiencing the same kind of data anomaly, the fact that Weekly_Sales is upstream of Revenue is useful information for deciding where to start.

Ensure heavily-referenced objects are monitored

Data engineers hate the dreaded, “Hey, why is this dashboard broken?” message from their data stakeholders. But because it can sometimes be hard to know exactly how many people are using a certain metric, Metaplane makes it easy to see which objects have downstream lineage in the app, and even—based on the shade of the indicator—how much downstream lineage that object has relative to the rest of the database.

We also provide helpful hints on the lineage visualization itself. From any table, view, or column, you will be able to see how many monitors that object has.

These hints help you spot holes in your monitor coverage, and helps ensure that important upstream objects aren’t forgotten about.

Raise data awareness

Onboarding as a new member of a data team—or even simply starting a new project in a part of the data stack you’re not familiar with—can be daunting. It takes a massive amount of effort just to understand how data flows through your systems and what dependencies exist. The new lineage visualization provides a holistic picture, giving a bird’s eye view of your data from ingestion, to warehouse, to transformation, and finally, to your business intelligence tools.

The visualization also helps empower teams to answer their own questions. Say a software engineer wants to know what downstream dependencies might exist for a table, or a data analyst wants to know which tables or columns power a dashboard. Instead of needing to message a data engineer with those questions, those individuals can explore the lineage visualization for themselves.

We’d like to leave you with an anecdote from Andrew, a data engineer at Reforge, about his experience with the lineage map:

❝A PR reviewer suggested removing some columns from an analytics table, but Metaplane’s lineage map showed me that one of the columns was being used on a critical company-level dashboard.
‍
As a data engineer, I’m constantly asking myself how changes I make in the data transformation layer will affect the BI layer downstream. The lineage map has quickly become the tool I reach for to answer those questions."

We hope that, much like Andrew, our new lineage visualization gives you the capability and confidence to explore your data stack with ease. This feature is live for all Metaplane customers.

Not a Metaplane user? Give Metaplane a try today.