Introducing source-to-target monitoring: Keep your data consistent throughout your pipeline

Most data quality issues start at the source. This might make solving data issues sound easy—just head to the source!—but most organizations don’t have just one data source.

Data is flowing from loads of different sources, making it tricky to make sure the data arriving in your warehouse is consistent with the source it came from.

Data consistency has real business implications, too. Imagine an e-commerce company putting in a rush order for added inventory because their stock levels were low, only to find out it was actually a sync issue between the warehouse management system and the CRM.

That exact challenge is why we’re excited to introduce a new tool within Metaplane: source-to-target monitoring.

With this new feature, you can compare data from multiple data sources in one monitor, so you can be confident that your data stays consistent throughout your pipeline.

Meet source-to-target monitors

With source-to-target monitors, you have an automated way to validate your data at every step of its journey. By defining a source metric, a target metric, and a comparison rule, you can ensure data consistency with minimal effort.

These monitors can be configured for:

• Row counts: Do source and target contain the same number of rows?
• Freshness: Is your target data up-to-date?
• Cardinality and custom SQL: Compare unique values or define your own metrics.
• Group-by comparisons: Validate data consistency across specific dimensions, like regions or product categories.

All of this is powered by Metaplane’s machine learning-powered anomaly detection, but you can manually configure thresholds for added control.

How source-to-target monitors work

Setting up a source-to-target monitor can be done in minutes. Just like the standard monitors you know and love, source-to-target monitors will automatically learn the behavior of the data, incorporate seasonality trends, and detect anomalies over time.

By default, source-to-target monitors will monitor the difference in measured values between your two assets. However, you can also monitor percent difference, which can be particularly handy for smaller tables.

Source-to-target monitor FAQs

What databases are supported?
All databases supported by Metaplane can run source-to-target monitors.

Can I use these monitors within the same database?
Yes, source-to-target monitors work seamlessly even when comparing data within a single database.

Do I need to write SQL for every monitor?
Nope! You can use our no-code interface and select from existing monitor types.

Ready to Get Started?

Read our docs and head to the Metaplane app and set up your first source-to-target monitor today. 

Or, if you’re new to Metaplane, book some time with our team. We’d love to show you how to use our end-to-end data observability platform to boost data quality and data trust within your organization.