Why creating metrics is so hard

Kait Schorr
By
Kait Schorr

The modern world of business is increasingly complex and competitive. Companies need every edge they can get to succeed in this climate and that makes it critical that business decisions be thoughtful and well informed. But how do we inform those decisions? The easiest answer is ‘data’, but what data and how?

Data is nearly useless in its raw or uncleaned state, and to derive any value from it we need to refine it into useful forms. Enter: the business metric. In the simplest terms, a business metric is a number that has distilled a much larger amount of data into a form that is understandable and actionable by a business leader.

The benefit of business metrics

Before a company invests a lot of time, energy, and work into developing business metrics, it’s important to agree and understand why they’re necessary (beyond just a vague notion that it will help guide the business).

There are three main areas that benefit from having and using metrics:

1. Business growth and scalability

As businesses expand, the need to monitor performance and scale operations becomes more and more important. Metrics provide a snapshot of where the company stands and highlights areas for potential growth as well as those in need of attention. Also, understanding the trend of a company’s growth is crucial for securing funding and managing the risk around financial decisions.

Questions answered:

  • Should we expand right now? If so, how aggressively?
  • How can we improve our operational efficiency to handle increased demand?
  • What are the potential risks associated with our growth strategy?
  • What marketing strategies will be most effective in driving business growth?

2. Monitoring and improving performance

Without a way to record and compare, there is no way to really know if a business or department is improving. Metrics allow for establishing a baseline of performance, and then comparing to that baseline over time to evaluate the effect of business decisions.

Using metrics within a business (such as KPIs or OKRs per department) also provides clarity and accountability to the organization. Knowing the metric a department is being measured by provides a concrete goal for to strive towards and allows for quantifiable achievements to be celebrated.

Questions answered:

  • Which areas or departments have seen a decline in productivity, and what factors are contributing to this?
  • How satisfied are our customers with our products or services? Is that improving?
  • Which sales strategies are most effective, and how can we optimize them?
  • How did a big decision/change we made last year effect our current bottom line?

3. Decision-making

Ultimately, the biggest reason to have metrics is to guide decision-making. Monitoring progress isn’t useful if that monitoring doesn’t yield any action. Having hard data that says “this is the right path” or “this will improve retention” is critical to making sound decisions and avoiding costly misdirections. Those same metrics can be used to provide a feedback loop that confirms the wisdom of previous decisions and builds the foundation for deeper understanding of business needs.

Questions answered:

  • Should we discontinue this product?
  • Which ad platform should we focus our attention on?
  • What is our ideal customer profile?

The challenge of metric design

It is estimated that 328.77 million terabytes are being generated everyday in 2023, and that number is only expected to grow. Companies are generating so much raw data and hoping that it will eventually translate into information they can use. Gone are the days where a simple question like “how many customers do I have?” could be answered with one data source and a COUNT() function. Most businesses are grappling with combining data from 4+ sources and struggling to clearly identify the metrics they want to pull from those sources. Why is defining a metric so challenging?

The answer to that question ultimately lies in the gaps shown here:

Metrics start with a broadly desired measure, such as “How is the business doing?” To get at such a broad question, we narrow down to a more technical and measurable question such as “How many customers do I have?” From that point we have to settle on what we consider to be an acceptable answer and/or how we plan on calculating it. In between each of these steps we have gaps. Each of these gaps represents a level of ambiguity or uncertainty that can be minimized, but never eliminated.

Gap 1: The business question gap

In the first gap, the ambiguity is around whether the question we’re asking actually addresses what we’re really looking for.

For the above example, we’re trying to measure how well the business is doing. So we want to count how many customers we have, right? That seems like a good number to show business success. But is it? What if we go from having 100 customers that each spend $100, to 1000 customers that each spend $10? By our customer count metric, we’re thriving and the business is growing. The revenue, however, tells a different story. Customer count can still be a useful metric, but we need more complex metrics, or combinations of metrics, to get closer to the truth.

Gap 2: The definition gap

In the second gap we get the uncertainty of definitions. The concept of a customer does not have an absolute definition, we have to make one. Is a customer anyone who has signed up on our site? Anyone who has spent money? Anyone who has spent money or started a trial with us? Anyone who has spent more than $1? It may seem pedantic, but these questions matter and have cascading implications.

Adding to the difficulty of defining things, we have different departments with different concerns. A customer for the marketing team may not equal a customer for the RevOps team. There is no perfect definition, and there will always be edge cases that need to be accounted for. Even the simple definition we’ve settled on here “Anyone who has given us money” yields questions. What about people who have made a purchase and had the entirety refunded? They’ve technically given us zero after the refund, but they performed the act of giving us money. What’s most important here is that a definition is agreed upon, and someone owns either answering these edge cases or the business accepts the level of uncertainty from an overly simple definition.

Gap 3: The technical implementation gap

In the third gap we have the limitations of data. Even with a very precise definition of what we’re wanting to measure, that measurement is only as good as the data we have at our disposal. Maybe we switched billing systems and now some customers are duplicated. Maybe we got a number of GDPR requests and the way we processed them was deleting customer records entirely. Maybe a customer accidentally created two accounts under two different emails, so they are technically one person but show as two customers in our metric. There are things we can do to address data issues, but there will always be real world limitations that prevent our metric from being 100% accurate.

Addressing these layers of decision making that go into a final metric is the biggest source of difficulty for metric design. It requires a deep understanding of both the real question being asked, the available data from which the metrics can be constructed, and the limitations posed by the business systems available.

How we can help

Well defined business metrics are critical for any business operating in today’s complicated world. Having solid metrics helps a business grow, make decisions, and monitor its progress. Defining those metrics, however, presents a real challenge. It requires a lot of decision making and clarity on what the business wants to know and how things are defined. Even the best defined metrics may not fully answer the broader questions that plague a business, but there are ways to minimize the complexity and challenge of creating metrics.

At Preql we’re working to reduce the ambiguity and complexity of building business metrics from multiple data sources. We’ve lived in this space and felt these struggles. We’re determined to help close the gaps by:

  • serving your core business metrics from all your data sources in one place
  • increasing transparency around how they’re defined, and
  • making it easier to consume your metrics in Preql or by integrating with your BI tools

If you want to learn more about how we can alleviate some of your metric-induced headaches, book a demo today!