Integration update: deeper observability into dbt

There’s one question related to dbt that we hear often at Metaplane: how should I think about dbt tests versus Metaplane monitors?

From our perspective, there’s no “versus” about it. Both are important parts of a modern data stack, and have their own individual strengths:

• dbt tests are ideal for situations in which you have hard constraints for the data (example: this column should always be unique). They work well when any issues with those constraints should result in a full stop of your transformation process so that your team can investigate it.
• Metaplane monitors are best when you want ML to detect anomalies due to nuances in seasonality, ingestion patterns, and so on. They can provide fuller coverage across your data stack, rather than focusing on just dbt models.

Nearly all of our customers who use dbt have at least some dbt tests.

But managing both at once can be a complex job, which is why we’re delighted to announce that you can now see additional metadata, including information about your job runs, model durations, and dbt tests, all in one place. Our users have long asked for a “single pane of glass” when it comes to dbt metadata and functionality, and with this update you’re now one step closer to having a one-stop shop for all your data observability needs. 

With this update you can now:

• See durations and statuses for all dbt job runs: Metaplane has long enabled you to create machine learning-based monitors to alert your teams to anomalous job durations—but now you can see the duration and status of any dbt job, monitored or not. Jobs that are completed in a shorter amount of time than normal might indicate incomplete data while jobs that are completed in a longer amount of time might be responsible for downstream staleness issues. In any case, having this insight helps you quickly eliminate potential causes from your troubleshooting list.
• See dbt test results: See the results of each test run on the corresponding job and model pages in Metaplane, enabling you to dig more deeply into how test failures may be impacting other metrics or downstream assets.
• Better understand root cause of data incidents. You can now see recent dbt job and test runs related to the data involved in an incident, allowing you to dig into what may have gone wrong with more speed and less frustration.
• See trends in model runtimes. See how long individual models took to run in any given job run, allowing you to spot issues or trends with individual models over time.

Viewing dbt test failures at scale

Integrating dbt events into the Metaplane platform also gives users the ability to view your history of dbt test failures. On any given dbt model, you’ll be able to see the results of your dbt test runs, as well as the objects that failures are associated with.

Additional incident management tools from Metaplane

In addition to the added context from those dbt events, Metaplane offers a few additional features to make it easier to triage and fix data quality incidents.

• Monitor notes and audit history: You’ll be able to see when a monitor was changed, when incidents last happened, and see any previous notes on resolving the incident(s) or why a monitor was created. 
• End-to-end column-level lineage: Expanding on the list seen above, you’ll be able to understand the full chain of lineage from your ingestion tools to your business intelligence tools. Upstream lineage can help accelerate further troubleshooting while downstream lineage helps with identifying impact for prioritization of each incident.
• Tags: these can be imported from dbt or manually defined in Metaplane and can be used to handle everything from custom dashboards for a quick overview of your data health to alert routing for specific teams or severity levels. Tags ultimately help focus your data teams efforts on the most pertinent issues.

Getting started with Metaplane and dbt

If you already have a Metaplane account, you’ll need to connect dbt through an API key and/or commands in the CLI to sync lineage from your instance. 

Otherwise, to get started with Metaplane, you can create an account or pick a time to learn more about data observability best practices from the team. New users can setup connections and implement data observability monitors within the hour!

‍