New feature: monitor recommendations

Think about your data stack. From raw data and mart models to BI dashboards and reverse ETL connections, a mature data warehouse can have thousands upon thousands of discrete objects. Tables, columns, views, workbooks, pipelines, models, jobs—all of them managed by your team, and all of them linked together in complex and interconnected ways.

With so much to manage, how can you possibly figure out which things your company should be monitoring, and what sorts of issues you should be monitoring them for?

The easy answer is that you just monitor everything for everything, making sure that all the unknown unknowns are covered. Row count and freshness for your tables and views? Check. Uniqueness and nullness on id fields? Check. Mean, median, and standard deviation on anything numeric? Check, check, and check.

But…your team doesn’t really care about everything, do they? Should you really be monitoring models that are only upstream of a dashboard that no one in your company looks at anymore? Or tables that you’re about to deprecate? Or some columns everyone thought would be important but no one actually uses?

Monitoring everything for everything inevitably drives tons of alerts about unimportant data issues. And a tsunami of alerts results in alert fatigue, and real data issues getting missed. 

Metaplane’s worked with hundreds of data teams over the years. Through those conversations, we’ve built an intuition of what assets are most important to data teams and where data quality incidents are most likely to originate. Our new monitor recommendations maps that intuition to your individual data stack to recommend where to add monitors and what kinds of monitors to add, making data observability easier than ever to set up.

Some examples of Metaplane recommendations:

• Landing zones: We’ll recommend monitors for freshness and row count here to ensure that you’re always getting the full data at the times that you expect.
• Tables upstream of frequently-viewed dashboards: To eliminate stale or partial data as a root cause of your incident, we’ll also recommend freshness and row count monitors here.
• Columns used in group by statements: Because you’re likely expecting a predominantly static set of dimensions when using group by statements, we’d recommend cardinality monitors here to alert you to a new value being added to your set. We’ll also make sure that your group by statements are referencing fields with data through our nullness monitors.
• Columns used in JOINs: Your keypair columns require something to join on, so we’ll always recommend nullness monitoring here. In the event that you’re tracking common one-to-one or one-to-many relationships, we’ll also recommend uniqueness monitors for these fields.
• And more coming very soon, including monitoring for your most-queried tables.

How to fine-tune recommendations

These recommendations go a long way towards highlighting the most common brittle points of your infrastructure. Of course, you may want to fine-tune Metaplane’s recommendations even further, such as: 

• Applying those monitors only within a particular database or schema
• Targeting tables with particular tags (such as ‘critical’ or p0)
• Only monitoring tables or columns that match particular naming conventions (such as nothing with ‘_staging’ in the table name)

Once selecting a recommendation, you can easily pare them down to just the objects you care about most, reducing any potential for alert fatigue.

Metaplane’s monitor recommendations, when paired with its powerful machine-learning monitors, make getting started with Metaplane a truly out-of-the-box experience.

Get started with Metaplane

If you already have a Metaplane account, you can find monitor recommendations in your account today!

Otherwise, to get started with Metaplane, you can create an account or pick a time to learn more about data observability best practices from the team. New users can implement data observability within the hour!