New feature: Data insights

Mature data organizations can have thousands to tens of thousands of queries running per day. In addition to one-off queries used for testing and addressing ad-hoc questions, organizations frequently have huge numbers of recurring queries. These queries come in the form of data cleaning, aggregations, and other transformations required to progress data from its initial ingestion state to a format that’s more useful for the business.

Over time, however, you may start to notice that your queries are starting to run more slowly, causing ripple effects due to latency dependencies. Or perhaps your organization has grown more conscious about how it’s spending its money, and has asked the data team to do some warehouse cost cutting. Both of these needs can result in data teams scrambling to understand where and how to get the most bang for their optimization buck.

Data Insights Dashboard

Metaplane’s latest feature, Data Insights, makes it easy to understand where your team can make the highest impact when it comes time to optimize your query performance and cost. We do this by:

• Giving over overview of cost over time to help you understand at a glance whether optimizations are needed‍
• Highlighting where your costs come from–these can be viewed either at either a query or user level, including service users. This can help make your TCO more obvious.‍
• Highlighting your least performant queries based on query runtime

These insights are designed to be actionable, and you can explore each insight more deeply from the drill-down view. From that view, you can:

• Filter the data: Look at the queries generated by, for example, a single user or all of your service accounts. You can also change which time window you’re looking at, depending on if you’re looking to see larger trends or focus in on a specific moment in time.‍
• Group the data: Group by type of query—such as a table creation statement—or specific tables and roles that you want to understand cost or performance for.‍
• Aggregate the data: Sort queries to drill down into by total number of runs, credit usage, bytes, and other common performance metrics in your warehouse. These options provide a wide range of possibilities to understand and optimize your queries for both immediate gains as well as future incremental gains.

Query Explorer

Oftentimes there are insights you can gain by digging into the specific instances of each query, and you can easily do that from the Query Explorer. The Query Explorer shows all recent query runs for a particular query, including how much each query cost, how long it took to run, the warehouse and warehouse size it ran on, which user ran the query, and so on. From this view, you can also see daily trends over time in cost, runtime, and more.

Table-level data insights

As part of this update we’ve also overhauled the insights you can see about a particular table. Simply navigate to the “Data insights” tab for any table or view in Metaplane to see information about the query volume, the users and roles that most frequently query that table, as well as the slowest and most common queries run on that table.

How Metaplane Users Get the Most out of Data Insights

Metaplane customers are already seeing enormous value from the Data Insights dashboard:

• One customer has used the query explorer to look at spillage to local disk for their dbt queries. This helped them determine which models may need to run on a bigger warehouse.
• Another was surprised to find that their longest-running query took over 6 hours. By digging into what else was going on in their warehouse around that time, they were able to zero in on a queueing issue and fix it.
• Customers using BigQuery as their warehouse were able to gain deeper insight into terabytes billed and slot usage for different types of queries, giving them deeper insights into the biggest drivers of spend.

We’re excited to hear how your team uses the Data Insights dashboard to optimize your own data stack!

More to Come

We’re just getting started with our Data Insights dashboard, and have many near-term updates planned, including an easy-to-use roll-up of what tables and views aren’t being used in your warehouse, and may be good candidates for cleanup.

If you have any feedback for other insights you’d like to see, please reach out via the in-app chat feature!

Getting started with Metaplane

If you already have a Metaplane account, you’ll be able to find data insights along the navigation bar on the left-hand side of your screen.

Otherwise, to get started with Metaplane, you can create an account or pick a time to learn more about data observability best practices from the team. New users can implement data observability within the hour, including these new data insights to improve query efficiency and optimize cost.