A single source of truth for code changes with Metaplane Data CI/CD

The phrase “An ounce of prevention is worth a pound of cure” is something we—as data teams—know all too well. Preventing problems from ever happening in the first place is ten times easier than detecting a problem and triaging it after it’s happened.

Of course, not all data issues can be prevented. More often than not, the majority of issues are caused by upstream source systems.

But those problems we can prevent, we should prevent.

Prevention > detection

That’s why we launched Data CI/CD last year. Our CI/CD integration proactively prevents data quality issues in pull requests by identifying how your PR impacts both downstream assets and the data itself.

Our customers loved shipping fast without breaking things. But they wanted to ask follow-up questions like:

1. How have my tests been trending over time?
2. From a CI/CD comment, how can I dive into more details?
3. From Metaplane, how can I inspect the actual code changes of a failing test?

Bridging code and data

Answers to these questions exist, but are scattered across tabs. PRs live in your version control tool; metadata lives in Metaplane. To bridge that gap, we shipped a single source of truth for our Data CI/CD so you can:

1. View a historical record for all regression tests and PRs
2. Link to Metaplane from those PR tests
3. Link out to GitHub from Metaplane

A historical record of code changes

Now, when you integrate your GitHub account with Metaplane Data CI/CD, a full history of your PRs is loaded into historical metadata in Metaplane. For each PR, you’ll see the status of data regression tests, the downstream impact of changes, and how models are resolved to tables/views in Metaplane lineage.

When data quality incidents are opened, you’ll also see context about which recently merged PRs are associated with that data incident, so you can quickly check if a code change was the culprit.

The end result is that the metadata of your code changes and data changes are combined together in the Metaplane CI/CD page.

Bi-directional linking

All PRs in the Metaplane CI/CD page link out to the relevant pull request in your version control system, so you can dive into the complete code change details. 

The reverse is also true. Our CI/CD tests leave simplified comments that describe how downstream assets and data are changed by each PR. Now, you can expand into the full comment (which could include hundreds of assets) to understand the full impact.

Get started with Metaplane Data CI/CD in < 5 minutes

Prevent yourself from making breaking changes before merging any new code (and save your team time, resources, and headaches) with our improved Data CI/CD feature.

If you already have a Metaplane account, you can get started with Data CI/CD in less than 5 minutes. Just check out our docs to set up dbt (cloud or core), connect GitHub, and configure your CI/CD integration.

Want to get started with Metaplane for the first time? You can create an account for free or pick a time to learn more about data observability best practices from the team.