Investigate bottlenecks at the model level with dbt observability

dbt is like a data team’s circulatory system: it keeps everything in the data pipeline moving smoothly.

When all jobs are running normally, your data flows to all your downstream tools on time. Everything is as it should be.

But when those jobs start running longer than usual, it’s like a clogged artery in your once pristine pipeline. Blood is still moving, but it’s slower, and you start feeling the effects of the reduced blood flow in your other organs (except, in this case, it’s slowing down the rest of your queries and impacting everything downstream). 

If things get worse and these jobs fail entirely, the data team is effectively in complete circulatory failure. Nothing works and the entire pipeline comes to a screeching halt.

So regardless of whether you’re dealing with a slowly clogging artery or complete circulatory failure, you need to figure out what’s causing your problems—fast.

Know about long-running jobs (before they turn into failures)

That’s why we launched our dbt observability. The initial launch supported both dbt Core and dbt Cloud, importing dbt job lists and durations across all projects to alert on anomalous job durations.

Our customers—both Core and Cloud alike—loved the visibility, but wanted to go a level deeper into individual models, so they could answer questions like:

• Why did this job take X% longer than usual?
• Which particular models in this job contributed to that increased run duration?
• How can I automatically know if there’s a spike in my job’s run duration?

We decided to give the people what they wanted: a best-in-class observability experience for dbt down to the model level.

dbt makes metadata available. Metaplane makes it actionable.

dbt does a great job of making metadata available in their Cloud API, but that’s usually where it stays, untapped and underutilized. Metaplane ingests all that valuable dbt data to dive deeper than ever and offer model-level insights.

It’s like an angiogram for your data: it gives you a clear view of your data's flow and shows you exactly where the blockages are. You can see not just that a job is taking longer than usual, but precisely where it is and what might be causing it.

With our enhanced dbt observability, you can now:

1. View recent jobs and duration
2. Investigate long-running models
3. See historical trends of the query runs that power each model
4. Get alerted on any anomalous job and model durations

Root cause analysis at the model level

See detailed information on the models involved in the 10 most frequent jobs over the last 7 days. If a job takes longer than usual, you can click into it to find which tests and models were involved. 

Maybe a model is taking longer because it's processing more data than usual, or a test is failing and causing delays. Our machine learning algorithm highlights any unusually high job durations based on the average over the past week—so you can get right to the heart of the problem instead of spending hours trying to figure out what’s going on.

Full visibility, all in one place

Instead of jumping between different tools and dashboards, you have one central hub where you can see everything and trust that everyone on your team is looking at the same information.

Metaplane’s dbt observability also links to Data Insights, so you can easily access trends over time for the queries that power each model, identify patterns, and understand how performance has changed within each model query.

Forget correlating data from different sources. You have everything you need to quickly identify issues and understand their context in a single place.

Alerts you actually care about

dbt sends alerts when a job fails, but wouldn't it be better to get a warning before that happens? That's exactly why our machine learning algorithm sends Slack alerts when a job's duration falls out of the normal range: to give you a heads-up that something might be wrong.

Now, if a job that usually takes 5 minutes starts trending towards 10 or even 20 minutes, you can investigate before it fails entirely.

Get started with dbt observability in < 5 min

Be the first to know when something is off with your data and get started with our dbt observability in minutes.

Want to get started with Metaplane for the first time? You can create an account for free or pick a time to learn more about data observability best practices from the team.