Partner Highlight - Horizon Data

In August of 2021, Rijnhardt Kotze founded Horizon Data with the goal of building scalable data platforms for high growth startups. As the team and number of clients grew, Rijnhardt adopted tools like Metaplane to help his analytics engineers move quickly while still building reliable and maintainable data pipelines for their clients.

One of the biggest challenges for his team was getting ahead of schema changes and data quality issues. They often found out about issues from their clients, causing more time spent identifying bugs rather than building models to support their clients’ businesses.

After getting started with Metaplane in minutes, Rijnhardt introduced data observability to multiple clients so his team receives proactive alerts whenever data quality issues occur.

Moving from reactive to proactive

When the Horizon Data team starts actively developing for clients, the structure of the data warehouse can change every week, introducing an environment in which the behavior of data is difficult to monitor. Before Metaplane, it wasn’t uncommon for logical changes to break a client’s data model, but go unnoticed.

For example, making changes to models such as a join across tables led to hard to spot data quality issues such as volume changes or distribution drift. An issue like this could take Rijnhardt and his team days or weeks to spot, slowing down development time and causing clients to lose trust in the data.

To ensure they were the first to know about data quality issues, Horizon Data adopted Metaplane to catch data quality issues in minutes instead of days or weeks. As a result, Rijnhardt describes Metaplane to his clients as:

‍“Metaplane is the solution to finding out about data bugs before the CEO asks an awkward question about a weird-looking number.”

Bringing Data Observability To South Africa’s Most Innovative Airline

One of Horizon’s first clients to use Metaplane was LIFT, South Africa’s most flexible airline. LIFT is a customer-obsessed and agile airline that relies on data to run its demand-driven and efficient business operations.

LIFT chose Horizon Data to scale their data platform so they can continue to focus on what matters – creating a new kind of airline with customers at the center.

Rijnhardt realized the power of the Metaplane platform when it was able to proactively catch a dramatic decrease in rows due to an upstream modeling issue. As a result, Rijnhardt could see several tables that were impacted and quickly realized the severity of the situation.

The issue could have affected how channel partners and API integrators are paid, currencies are converted, and other business critical processes. The issue was immediately prioritized and fixed, saving days or weeks of issue diagnosis and recovery.

Monitoring for unknown unknowns

One of the challenges of building and maintaining data pipelines is that because the surface area is so wide, it’s hard to keep a pulse on everything that’s changing across these systems. Without continuous monitoring, it’s impossible to understand how all of the data in the warehouse behaves over time.

After adding Metaplane on top of LIFT’s data warehouse, Horizon was able to not only ensure trust in the data, but also identify changes in their business that would have gone unnoticed.

‍“Metaplane has solved data quality problems, made us aware of issues happening in the background that we wouldn’t have known about, and democratized the data that used to be in one person’s head.”

For example, as LIFT launched new services, Metaplane would alert Horizon about changes in the data. These changes, such as distribution changes in metrics about flights, could be communicated to LIFT’s commercial teams so they could understand changes in the business over time. If Metaplane was not actively monitoring the behavior of the data prior to these changes, it would be impossible for Horizon to identify these shifts.

Optimizing for efficiency

Horizon Data chose Metaplane so they could focus on creating value from their clients’ data rather than fixing data quality issues. One difference they noticed after adopting Metaplane was removing unnecessary meetings.

For example, before Metaplane, Rijnhardt found that many meetings with his clients would be concerned with confirming upcoming schema or logic changes to make sure there were no unexpected upstream changes from the data warehouse.

“With Metaplane, monitoring is automated in the background, helping our team focus on what they’re good at.” Not only does it help during development, but creating a more efficient process makes it easier to hand off projects when they are completed.

Horizon Data’s Best Practices

As Horizon Data has utilized more of the Metaplane platform, they have become experts in setting up a data observability platform for their clients. Rijnhardt recommends setting up Metaplane as soon as possible, including for new projects, to make sure that data quality and unknown unknowns are accounted for. In his own words, “adoption and awareness of Metaplane by management is worth it from day one.”

One reason Horizon adds Metaplane from day one is to take advantage of schema change monitoring. Since Horizon is often downstream from changes made by software teams, it’s difficult to keep up with the many upstream schema changes that frequently impacted the warehouse and reporting.

For example, if schemas changed in upstream relational databases, it would take days before Horizon would become aware, causing the team to reactively fix transformation or data integrity issues or find ways of modeling new data to support new use cases.

With Metaplane, they started receiving alerts as soon as schema changes were made, avoiding costly data quality issues and encouraging active communication with upstream teams. Rather than “knocking on the door of a product manager asking what schema changes were introduced by their software engineers”, Rijnhardt was proactively alerted about these changes.

Metaplane is excited to continue our partnership and bring a data observability platform to Horizon’s clients in South Africa and Europe. Not only has Horizon been instrumental in educating data teams on data observability, but they provide valuable feedback to our product team.

When asked about what he enjoys about partnering with Metaplane, Rijnhardt said he “appreciated having deep access to the team, helping us solve problems and creating a collaborative experience.”

* * *

Are you in need of a trusted data partner to help take your data stack to the next level? Get in touch with Horizon Data -->

If you are a data consultancy looking for a data observability platform to help your clients trust their data and get ahead of data quality issues, get in touch or become a partner by filling out this form.