To set the scene, the data engineering team creates and manages the data pipelines that handle everything from extracting and loading data from outside the warehouse to multiple levels of transformation before it’s exposed to a data consumer, such as in the form of a business intelligence dashboard.

The data teams’ end users include finance and accounting teams, loan servicing and origination analytics, and sales and marketing, but before it ever reaches them, data actually passes through a medallion architecture. In this way, data is safely processed from a raw state into a normalized form prior to being able to be used in production datasets.

‍

‍

All of this is to say: data is key to Upright’s strategic operations, but it wasn’t always this way. In fact, when Rebecca first started at Upright, the majority of querying was happening through an existing Metabase instance plugged into a replica application database.

Implementing Sigma to improve data exploration

At this point, the primary users of Metabase were engineers that interacted heavily with the application database, who would create reports to answer ad-hoc questions - to the tune of over 1500 different reports. As Rebecca and the data team were continuing to stand up Snowflake as the new analytics data store, they wanted to pick a new business intelligence tool that would:

• Continue to foster data exploration
• Expand the usability to audience(s) with varying degrees of technical skill
• Integrate with Snowflake
• Support guard rails to ensure that users were working with accurate, curated data

After evaluating other prominent business intelligence tools, they ultimately decided to go with Sigma, as it best met the requirements above. Fast forward to today, and the combination of Sigma accessing new data that Rebecca’s team has ingested into Snowflake has allowed for easier reporting than ever before.

❝For Jira analytics to understand engineering performance metrics, such as sprint over sprint comparisons for velocity and task fulfillment, we actually now create reports in Sigma, rather than Jira itself.

How Upright uses Metaplane

Data quality before Metaplane

As the team rapidly stood up new dbt projects and models to support the new objects that the data engineering team had imported, they were often also pairing them with dbt tests, to the tune of roughly 100 or so. Similar to most dbt unit tests, they were often checking for boolean outcomes - for example, whether or not primary keys in a table were unique.

As the team began to think about how to scale the tests for greater table and model coverage, along with results that could display a range of outcomes, they began to look into additional vendor solutions.

To complement the upstream (recalling the medallion architecture mentioned earlier) placement of dbt and corresponding tests, the team implemented Metaplane. The strategy created by Rebecca and her data engineering team was that dbt tests would serve to validate some data during the scheduled model run to create a safeguard against bad data persisting downstream through the warehouse into Sigma workbooks and that Metaplane would provide an additional layer of data quality assurance for production datasets.

Implementing Metaplane

From the start, the team immediately recognized the benefits of Metaplane’s machine learning based approach for calculating, setting, and updating acceptable data quality metric thresholds based on Upright’s specific data sets, down to table and field levels. 

❝The machine learning (in monitors) identifies trends for the team so it’s not something that they need to keep tabs on to update (for accurate incident alerts).

After proving out the value on the initial set of critical tables, Rebecca and her team went to work to set up monitors for: Column count, row count, freshness, dbt job durations, and some custom rules to boot.

Row Count

This is by far the most used monitor in Upright’s roster, with over 150+ monitored tables, including payment related, loan related, and 3rd party data. One part of why they’ve implemented so many row count monitors is to capture issues that have originated upstream out of the data engineering team’s purview. Some of examples of when this has happened include:

• Dummy account creation - Upright will occasionally get attacked by bad actors in the form of fake account creation to request loans that aren’t intended for them. Those fake accounts lead to a spike in the row count of the corresponding users table where remediating the issue leads not only to happier executives tracking user counts but also avoiding accidentally introducing money to someone that isn’t supposed to be getting it.
• Unintended effects of code changes - In one case, an update to code responsible for task creation, noted as rows in the upstream database, accidentally created tens of thousands of new rows that the data engineering team was able to flag to the software engineering team before any harm was caused. In this way, they were able to continue to be responsible neighbors and build trust

Freshness & dbt job duration

As we mentioned earlier, these monitors are used for production datasets, often modeled through dbt. As a result, both the dbt jobs themselves and the update cadences of the model outputs and related objects are monitored. While dbt itself is extremely easy to use, concurrent jobs will occasionally conflict, leading to unexpected latency. These monitors ensure that the downstream teams aren't working with stale data.

Custom SQL

One example of Upright’s data engineering team is protecting the business is shown in a custom monitor. Upright offers loans for investments, and naturally, have set timelines for users to pay back the loans. They have one monitor calculating the number of loans with negative remaining months for paybacks, which ensures consistency over how a loan is represented over 3 disparate systems. 

This monitor has led to a smoother working relationship between the data and software engineering teams by allowing the data engineering team to more easily raise bugs when data is mismatched between those systems.

Suggested Monitors

To understand where to add new monitors, Rebecca and her team are also using Metaplane’s Suggested Monitor feature.

❝As an engineering manager, I don’t really have time to even do much hands-on coding. Being able to know where to just switch those things on through Metaplane is very nice.

Troubleshooting data quality issues with column level lineage

While Upright already had dbt that generates lineage diagrams, Metaplane was able to expand on the usage of lineage graphs to include other tools beyond dbt itself. Without the lineage graphs, the team used to have to take time to actually scan the code itself, which started with understanding which table(s) a Sigma workbook was referencing, connecting those dots, using dbt lineage to source it back to an origin table(s), and then testing those tables in the warehouse.
With Metaplane, they’re now able to understand all of the object dependencies from Fivetran down to Sigma to easily understand the scope of any given issue.

❝Because I have so many systems to observe at any given time, I default to going to Metaplane's lineage graph because it gives me that end to end view. Having Metaplane connect the dots between Fivetran down to the Sigma report that’s getting flagged has proved to be an invaluable resource. Metaplane has short circuited the triage process from a few hours down to a few minutes.

Preventing data quality issues with data CI/CD

Rebecca’s data engineering team continues to make strides in improving their data modeling. As responsible data modeling code owners, they’d like to avoid creating any issues with updates to these models. Using Metaplane’s Impact Analysis and Test Preview features which allow them to not only see which tables, models, and Sigma workbooks would be impacted, but also show how data will be affected.

❝Metaplane’s Github application has been indispensable as a tool for additional validation as we change code in our dbt instance. It makes it super simple to understand the downstream impact of any sort of change, and when paired with lineage, we can go directly to analysts to tell them ‘X change has Y impact’ to prevent accidental issues

Looking up for Upright

Building upon the success of Metaplane to date, Rebecca and the Upright data team want to open up Metaplane usage to the analyst team, so that they don’t need to rely on materialized views of dbt tests to check for ad-hoc issues. The expanded usage will also include more custom SQL tests to reflect unique business constraints.

As we mentioned earlier, as FlipperForce and FundThatFlip continue their evolution into Upright, they’ll also need to merge their data efforts, which Rebecca’s team will play a large role in leading!

‍