M3: The Metrics Maturity Model

Successful founders, looking back, often talk about how they used data to track their progress and inform their decisions.

But how did they get started?

During the early days of any new venture, it’s easy to feel completely overwhelmed or outright misled by data. Even when we understand the importance of looking at our numbers constantly and sharing regular updates, starting with a completely blank spreadsheet is a daunting prospect.

So don’t panic if you feel like that. It’s completely normal.

Using data like this is a behaviour that we can learn. There are multiple levels and we can progress through them as we go.

Level 1: Learning Russian

In the very beginning all we have are numbers.

The best analogy I have is the science fiction nightmare of waking up to find we’re locked in the control room of an abandoned Russian nuclear power plant that is just about to melt down.¹ We can see the control panel, with all the dials and switches, and we know we need to do something, but none of the labels make any sense.

This obviously doesn’t apply to any readers who already speak Russian and so would cope better in this scenario. Please substitute whatever language would be most indecipherable to you!

As in the meltdown scenario, so in the fledgling startup. We’re rapidly learning which lever is which by trial and error. It’s impossible to know exactly which of the numbers we have are going to matter most to our venture as we grow, so we need to collect them from everywhere – our website analytics, our finance and billing systems, our customer support systems, our sales pipeline, etc – and get them all in one place as a foundation to build on.

Actions:

1. Make a list of the various systems that are collecting numbers about our venture: our finance or billing system, our website analytics, our customer support system, our sales pipeline system, etc.
2. Then, collate everything in one place (a shared spreadsheet is a good start). This will give us a foundation to build on when we come to more detailed analysis.

Level 2: The habit of regular updates

Once we have the numbers at our fingertips it’s tempting to jump straight into analysis. But wait! It’s not the complexity of the maths we do or how many three-letter acronyms we have that matters most, but the consistency of the habits formed around regularly updating our numbers, looking at the trends and sharing them with somebody who can help us decipher them. Maybe we can do that all by ourselves, but more likely we’ll lean on a team member, an external investor or an advisor. It’s harder than we ever realise to punch holes in our own reality distortion field. The goal is a regular cadence.

It’s shocking how many founders never build this habit. These days the best sign I have that a startup in which I’ve invested is dead, or near death, is the silence. Despite starting with the best of intentions, gradually the frequency of shareholder updates starts to dwindle. Why does this happen? Sometimes it’s because our numbers are not in a format that is easily shared. More commonly, we’re embarrassed because the results aren’t what we’d hoped for and confidently predicted they would be. That’s understandable. The natural inclination is to wait until the numbers are better. Maybe we can fix things before anybody notices? But when we do that, we keep information out of the hands of the very people who can help the most. As soon as we know, or even as soon as we have a sense that things are off-track, we need to say that out loud. Problems are so often easily fixed – much more often than we fear when we hesitate to ask for help, and especially when we catch them early.

This doesn’t have to be complex or time consuming. We can start by sharing the most basic financial details: how many new customers we have, how much we earned and spent in the last month and how much cash we have left in the bank. Later, as the business grows, we can include some commentary about the current constraints. Start with the question: why aren’t we growing faster? Talk about people: new hires, open roles, how we’re feeling about the internal culture. Highlight the key numbers: how much are we spending to acquire each new customer, how much does it cost to support each customer, and how many customers cancel (or “churn”) each month? Ask others in the team to talk about their area of the business. These are all details we should have at our fingertips now (see Level 1 above). The details we share with our team and with external investors and advisors can just be a subset of reports we create for ourselves.

The goal is: simple updates on a regular cadence.

The most important thing is to be consistent and regular rather than occasionally amazing but neglectful the rest of the time. I’ve learned that an even bigger payback comes when we want to raise additional capital in the future. There is nothing worse for an investor than an email from out of the blue asking for more money. If we’ve taken the time to keep everybody informed then we’ll spend much less time telling them where we’ve been, we can focus on where we want to go next, and we’ll likely find investors are much more enthusiastic about continuing to be part of that journey.

So even when we think we have nothing interesting to say, say something.

Actions:

1. Choose at least one other person who is interested in our progress and share our metrics with them on a regular basis. This could be as often as weekly in the very early stages, but at least monthly as we grow.
2. Ask them what metrics they think are most important and what they would consider good progress to be at our age and stage.
3. Ask them what other metrics we should be tracking that we might currently be overlooking.

Level 3: Crossing the streams

Once we have a handle on the basic metrics associated with our business, and we’ve formed the habit of updating and sharing them regularly, we can be more savvy about our analysis. Often the most interesting metrics are not the things that we can record directly – such as the number of unique visitors to our website, the number of paying customers, or the revenue earned – but the combinations, derivatives and countermeasures.

Combinations

Ratios are a more nuanced way to measure our progress than the raw numbers, because they are more easily compared across different time periods and to industry benchmarks. For example, rather than looking at the number of new customers we gained or how much we spent on advertising, combine those to calculate our cost of acquisition (often abbreviated to CAC) – that is, how much we spent on average to acquire each new customer. Or, for any kind of business that is not yet profitable, rather than recording how much cash we have in the bank, and how much cash we are burning (total expenses net of revenue), combine those to calculate our runway – how many months we can continue in the current mode before we run out of money?

Derivatives

We can also calculate our growth and acceleration. For example, if we sell subscriptions, in addition to measuring our average revenue per subscriber (often abbreviated to ARPU), calculate the change – is the ARPU increasing or decreasing? Often, to get value from derivatives we need to look back over a few periods, so we can really understand the trend and avoid being distracted by one abnormal data point.

Countermeasures

Whenever we identify a positive metric, it’s useful to ask: what are the downsides? When we focus on one thing, such as growing the number of customers we have, we risk losing sight of the collateral damage this could cause in other areas of the business. Having a good countermeasure will ensure that we have healthy growth. A good way to identify countermeasures is to put on our tinfoil hat and ask: if somebody in the team was maliciously trying to juice this metric, how would we tell? For example, if we measure the average time it takes to respond to a customer enquiry then, as a countermeasure, we should also look at the quality of the responses. A customer service team that is hyper-responsive but fails to solve the problem the customer is experiencing is no success.²

One key metric for Trade Me was the number of active listings. But we also had to pay attention to the quality of listings in each category. We tracked the ratio of sales to inventory (often called the sell-through rate), or the ratio of sellers to inventory (the listings per seller), as a way of checking that we had valuable listings and not just lots of them.

Actions:

1. Look for combinations or ratios, then compare these to industry benchmarks.
2. Look for derivatives, to demonstrate growth.
3. Look for countermeasures, to avoid unintended negative consequences.
4. Keep sharing results with anybody who can help us understand them better.

Level 4: Lines in the sand

The next step is to create feedback loops, so that we can constantly improve. We can choose a handful of important metrics and set a target for each – either a level (more than 100 new customers this month, say) or a range (average sales per day of between $500 and $700). These could be driven by the business model, the cash position or just arbitrary growth objectives. Either way, we should try to be realistic about how quickly and how impressively we can make changes and achieve these results.

Once our goals are defined, whenever we update our metrics we can compare our actual performance to where we imagined we would be, and think about the gaps. If we can explain why we did or didn’t hit the target it’s useful to write those reasons down. This is a form of debrief, and if we do it regularly we’ll get much better at it. Remember, the questions to ask are: what went well and what didn’t, what surprised us, and what did others do in response? Then we can look back later and ask if they stood the test of time, are recurring excuses (which is a problem) or something we’ve subsequently fixed.

Lagging vs. Leading

Most of the things that are easily measured are lagging indicators, showing what has happened in the relatively recent past. It’s much more valuable if we can uncover a leading indicator – something that helps us to know that we’re already on the right track. Possibly the most famous example of a leading indicator is the engagement measure used at Facebook. They discovered that if a new user has connected with seven friends within 10 days of creating an account they will typically remain much more engaged as a user. That’s a hugely useful thing to understand, because the team can then focus on how to get as many new users as possible across that threshold.

Actions:

1. Set realistic targets for each key metric. Share these with advisors to make sure that we’re not being too optimistic or not ambitious enough.
2. Create a feedback loop. When we report our metrics, compare what we actually achieved to the original targets we set.
3. Update the targets based on what we’ve learned, then rinse-and-repeat!
4. Look for leading indicators that focus our efforts where they have the biggest impact.

Level 5: The One!

By now we should have a much better grip on which metrics matter the most, share these regularly with others and compare the actual results with predefined targets. These feedback loops narrow the long list of metrics down to a more focused basket that captures the most important aspects of the business and clearly shows how we are tracking. These are likely to be specific to our business model and stage, and will almost certainly change over time as old problems get solved and we grow into new areas with their own new problems.

We can strip our full list of metrics back to just those that indicate if what we’re doing at the moment is working – the things we can take action on – then obsess about how to improve them. Often the best metric to choose will be a composite that captures several different and ideally competing aspects of the venture in a single number.

At Trade Me this was the number of different people (what we called “unique sellers”) listing items for sale. We correctly guessed that if we could increase that then it would create a positive feedback loop: more sellers would mean more diverse items for sale, more bids, more successful auctions, more feedback placed and critically more word-of-mouth growth. We started tracking this metric daily and quickly identified a number of improvements we could make to increase this number – mostly by lowering the friction in the process of listing an item for sale and making it much more obvious.

Another more recent SaaS example which became a key part of both Vend and Timely board packs when I was working with those companies is Tomas Tunguz’s Cost of a Recurring Gross Profit Dollar metric (CRGPD).³ This is a great composite metric: it captures lots of different aspects of the SaaS business model in a single number (including the gross margin, acquisition cost and churn); it’s immediately meaningful - if it’s costing more than $1 to gain $1 of recurring gross profit that should straight away raise concern; and by tracking the number over time we can quickly see progress - a lower cost is better than a higher cost.

Actions:

1. Constantly ask: what is the most important metric at the moment?
2. Then, obsess about what we need to do to improve that number.
3. Share our thinking, and ask others to challenge us to think harder about how the things we’re doing will impact our key numbers.

Vanity vs. Clarity

A common measurement trap is vanity metrics. These are the metrics that make us feel good – the number of visitors to our website, likes on our social media posts, or mentions in the media. They favourably compare our performance to others, or to arbitrary benchmarks. They tell a positive story. For those of us who are competitive by nature, the comparisons baked into vanity metrics can be very motivating. But we need to be careful we don’t fall into the trap of comparing our insides with others’ outsides. For example, I found it confusing when people celebrated how quickly the team at Vend was growing. Knowing when headcount tips from being a vanity metric to being vital is one of the keys to successfully scaling a venture beyond the startup stage. But unless we know the revenue per employee a bigger team just means a bigger payroll and shorter runway.

We also need to be wary of vanity milestones.⁴ These make for good press releases, generate attention, and give us something to publicly celebrate. For example, winning awards, partnerships with much bigger companies (which rarely, if ever, translate into meaningful sales for the smaller partner) or attracting small amounts of additional capital from celebrity investors. But we need to remember correlation is not the same as causation. Lots of successful companies win business awards. However, winning business awards is only loosely related to the things we need to do, have and be in order to build a successful business.

Vanity metrics and milestones are invigorating but also flattering and misleading. Our metrics shouldn’t be results we report only when they are great, in order to get accolades. They should be the indicators that we use to determine what is and (often more importantly) isn’t currently working. We should prefer clarity metrics.⁵ These highlight trends in our performance and ideally help us decide what to do next. They are actionable and fundamental.⁶ For example, average sessions per user, average time spent on-boarding, churn/retention rates by cohort, revenue-to-acquisition-cost ratios, and so on. Clarity metrics are humbling, sometimes exhausting, and essential.

We don’t need another hero

It’s tempting to be completely dismissive about vanity metrics and milestones. However, we can use them to our advantage. The average finishing time for a marathon tracked on the Strava app is three hours 58 minutes and 25 seconds – a whisker under four hours. The technical difference between a 4:01 marathon and a 3:59 marathon is insignificant – just 2.8 seconds per kilometre (perhaps a few strides). But the psychological difference is huge.

In addition to the metrics we use to inform our actions, it’s useful to have some hero metrics to demonstrate progress to people outside of the company, and to generate buzz that might tempt some of them to get more involved. The best recent example I’ve encountered is an environmental restoration project on New Zealand’s Coromandel Peninsula, where the metric they talk about is “the number of complaints about bird song noise”. The people running the project figured that was a great way to show they were having an impact, and even more, to signal to everybody in the area what the real purpose of the project is – that it’s not just about planting trees and clearing traps. We just need to always keep in mind which metrics are useful because they make us (and our team) feel good and which are useful because they help us decide what to do next. Those are likely different metrics.

What are we waiting for?

They say that the hardest part of going for a run is putting on running shoes. Once we’ve done that then heading out the door is an obvious next step. And it’s the same with metrics. We can’t let a blank spreadsheet scare us from taking the first step. Once we get started we’ll find that each of the levels are a natural progression from the last.

So get going and perhaps one day we’ll be a successful founder looking back talking about how much data helped us understand our venture and track our progress.

Related Essays