Monitoring v2 - What's New

Welcome! You’ve probably noticed that your monitor page has had a bit of a glow up. That’s because we’ve recently rebuilt how Metaplane monitors your data, resulting in a shiny new UI and more power under the hood.

There are several features wrapped up into this update, including:

• An entirely redesigned monitor page
• More predictable alerting behavior, and
• Greater visibility into underlying data for Snowflake customers

Monitors, but better

We’ve made several changes to filter out the noise, streamline workflows, and add useful context to help you get the most of your monitors.

The first thing you’ll notice is the redesigned graph at the top of the page. We’ve done away with the traditional y-axis in favor of contextual axes that make the relationship between the historical data and the most recent data point much clearer. You can now see the exact values for Metaplane’s predicted range, and where the observed value falls compared to that range without having to do any mental math. And we’ve added an extra layer polish to make it easier to see exactly when ran, and when data points were marked as normal.

We’ve also added extra context to the table below the graph. Now you can see at a glance whether an observed data point was in the expected range, and if it wasn’t, whether it was higher or lower than expected. And we’ve improved the way that the table displays manual thresholds, making it easier to understand how your data has historically behaved.

You’ll also notice a new “Overview” section at the top right, which shows high-level information about the monitor, including the current status of the monitor and when it was last run.

Below the overview you’ll see the new streamlined configuration options. There’s now an explicit option to switch the monitor between automatic, ML model-based monitoring and manual, threshold-based monitoring. There’s also a new option which allows you to select a specific date to begin incorporating historical data into the model—useful if your data materially changed at a certain point, and you don’t want the model to use the old data in making predictions. These configuration options, plus all the ones you’re used to, are just a click away in a side menu.

More predictable alerting behavior

You’ll probably notice that the graph has a new 🎉 marker on it, illustrating the moment your account switched to the new monitoring system. You might've also noticed that the expected range (the light green area) on many of your monitors expanded shortly after the change.

In our old system, there was a difference between the range of values Metaplane displayed as “normal” on the graph and range of values that would actually trigger an alert.

The new system is simpler. The expected range on the graph and the range that triggers an alert are the same, resulting in slightly larger expected ranges on the graph. If you see value outside that green area, Metaplane sent an alert.

As a part of these changes we also removed the “Alert level” option which gave you some control over the range of values that would trigger a data incident. We plan to introduce a new way of fine-tuning the expected range for your monitors soon.

More visibility into underlying data [for Snowflake users]

Most companies choose not to monitor everything in their database—for good reason. Between the noise and the cost, it’s often simply not worth it. But that doesn’t mean that the data in those unmonitored tables isn’t important, or that you don’t occasionally want to dig into the metadata.

Now, for Metaplane users with Snowflake warehouses, we’ll be capturing snapshots of the freshness and row count of every table or view in your database. You won’t get an alert about any drops or spikes in this data. But if you notice something’s amiss and you want to dig in, the historical metadata is there for you to do so.

And if, at some point down the road, you decide you want to add a monitor to the data, you won’t need to wait for Metaplane to complete our standard 3-5 day training period. Since Metaplane already has the historical data, we can create the model and have you monitoring that table’s row count and freshness in moments.

Best of all, because we’re able to get this metadata through Snowflake’s information schema, this extra context won’t cost you a dime.

Accelerated development for monitoring capabilities

While there’s a lot to love about this update inside of the Metaplane app, the biggest update is under the hood. We’ve completely rewritten how we do monitoring to be more performant and more flexible. This means faster page loads with fewer errors. It also means that we’ll be able to accelerate our development speed when it comes to new monitor types and capabilities going forward, getting useful updates into your hands faster.

‍

More of a video person? Check out the video walkthrough below.