Use the Metrics Maturity Model to improve how we measure and report our progress.
Successful founders, looking back, often talk about how they used data to track their progress and inform their decisions.
But how did they get started?
Is using data in this way a behaviour that you can learn?
In my experience, during the early days of a new venture, it’s far more common for both founders and investors to feel completely overwhelmed or outright misled by data than to feel informed or driven by it. Even if you understand the importance of looking at your numbers constantly and sending regular updates, starting with a completely blank spreadsheet is still a daunting prospect.
So please don’t panic if you feel like that! It’s completely normal.
Let’s map out the steps we can take, on the path towards metrics enlightenment.
Start simply by asking what are the most important facts in your organisation and how many people know them
— Hans Rosling, Factfulness
In the very beginning all we have are numbers.
The best analogy I have is the science fiction nightmare of waking up to find we’re locked in the control room of an abandoned Russian nuclear power plant that is just about to melt down.1 We can see the control panel, with all of the dials and switches, and we know we need to do something, but none of the labels make any sense.2 All we can do is rapidly try to learn which control is which by trial and error.
It’s impossible to know exactly which of the numbers we have are going to matter most to our venture as we grow, so to get started the best we can do is just start to track everything we can, beginning with a standard template of metrics that are applicable to our business model.
Make a list of all of the various systems we use that are collecting numbers about our venture: our finance or billing system, our website analytics, our customer support system, our sales pipeline system, etc etc. Then start to collate everything in one place. This will give us a foundation to build on when we come to more detailed analysis.
Action: create a spreadsheet, and collect all of your numbers in one place.
I must write each day without fail, not so much for the success of the work, as in order not to get out of my routine.
It’s tempting at this point to jump straight into analysis.
But, wait!
Often it’s not the complexity of the maths or how many three-letter-acronyms we have that makes the difference - it’s the consistency of the habits that are formed early around regularly updating our numbers, looking at the trends and sharing them with somebody who can help us decipher them. Maybe we can do that all by ourselves, but more likely it will be somebody else on our team, or an external investor or advisor. It’s harder than we probably realise to punch holes in our own reality distortion field!
It’s shocking how many founders never get beyond this step - either because they don’t have numbers in a format that are easily shared or they are embarrassed by the numbers themselves (and then mistakenly think that the best fix is to keep quiet until the numbers are better!)
This is no different to the advice that is given to people who want to improve their diet or be more active. The most important thing is to be consistent and regular than it is to be amazing infrequently but neglectful the rest of the time.
Our diet is not something we’re “on”, it’s what we consistently put in our mouths everyday. And, likewise, our metrics are not results we report when they are great in order to get accolades, they are the indicators that we use to determine what is and (often more importantly) isn’t currently working.
If we exercise regularly it makes a big difference to our wellbeing and once the habit is established it’s much easier to do consistently.
Likewise, if we regularly review our numbers with somebody who can be objective it makes a big difference to our venture, and once the habit is established it will become something that we do intuitively.
Action: find somebody else who is interested and share your metrics with them on a regular basis - possibly as often as weekly in the very early stages, but at least monthly as you grow.
Once we have a handle on the basic metrics associated with our business, and we’ve formed the habit of updating and sharing them regularly, the next job is to be more savvy about the analysis we do.
Often the most interesting metrics are not the things that we can record directly - e.g. the number of unique visitors to our website, the number of paying customers we have, or the revenue we earned - but the combinations, derivatives and countermeasures.
Combinations
What are the ratios that measure our progress?
Often these are more nuanced than the raw numbers, because they are more easily compared across different time periods and to industry benchmarks.
For example, if we have an online retail business, rather than looking at the number of new customers we gained or the amount you spend on advertising, combine those to calculate our cost of acquisition (often abbreviated to CAC) - that is, the amount we spent on average over the last period to acquire each new customer.
Or, for any kind of business that is not yet profitable, rather than recording how much cash we have in the bank, and how much cash we are burning (total expenses net of revenue), combine those to calculate our runway - i.e how many months we can continue in that mode before we run out of money?
Derivatives
What are the first or second derivatives of our basic metrics?
For example, if we have a SaaS business, in addition to measuring our average revenue per subscriber (often abbreviated to ARPU), calculate the change - is the ARPU increasing or decreasing? Or, if we want to be really fancy, calculate the acceleration - is the increase (or decrease) itself increasing or decreasing?
Often, to really get value from these kind of metrics we need to look back over a few periods, so we can really understand the trend and we are not distracted by one abnormal data point.
Countermeasures
Whenever we have a metric that is looking really positive, it’s useful to ask ourselves: what is the countermeasure?
Often when we are very focussed on one thing, such as growing the number of customers we have, we can lose sight of the collateral damage this is causing in other areas of the business. Having a good countermeasure will ensure that we have healthy growth.
A good way to identify countermeasures is to put on our tinfoil hat and ask: if somebody in the team was maliciously trying to juice this metric, how would we tell?
For example, if we have a marketplace business, then one of our key metrics is the inventory we have (i.e. the number of active listings on a classified site). But, it’s not just the raw amount of inventory that matters, but also the quality. So, we might also measure the ratio of sales to inventory (often called the sell-through rate), or the ratio of unique sellers to inventory (the listings per unique seller), as a way of checking that we have valuable listings and not just lots of listings.
Or, a classic example, if we measure the average time it takes to respond to a customer enquiry then, as a countermeasure, we also want some way to measure the quality of the responses, so that our customer service team don’t just rush off poorly-considered responses, but actually try to solve the problem the user is experiencing.3
Once we’re collecting, analysing and sharing our metrics, the next step is to create feedback loops, so that we can constantly improve.
Choose the metrics that we think are most important and set a target for each of these - either a level or range. These could be driven by the business model, the cash position or just arbitrary growth objectives. Either way, we should try to be realistic about how quickly and how impressively we can make changes and achieve these results.
Once we have goals defined, whenever we update our metrics compare the actual performance to where we imagined we would be, and think about the gaps - where have we made improvements?
This is a form of debrief, and if we do it regularly we’ll get much better at it.
Some good questions to ask are:
We should write down these things as we review our metrics and consider why we did or didn’t hit your targets. Then we can look back at those in the future and ask if those reasons have stood the test of time, are recurring excuses (a problem!) or have been something that we were able to correct for subsequently.
Lagging vs Leading
Most of the things that are easily measured are lagging indicators, showing what has happened in the relatively recent past.
It’s much more valuable if we can uncover a leading indicator - i.e. something that helps us to know that we’re already on the right track.
Probably the most famous example of a leading indicator is the engagement measure used at Facebook. The growth team there discovered that if a new user has connected with seven friends within 10 days of creating an account they typically remain much more engaged as a user. That’s a hugely useful thing to understand, as it then focusses the team on how to get as many new users as possible across that threshold.
Action: set realistic targets for each of your key metrics and check back in future months to compare what you actually achieved to the original target.
By now we should have a much better grip on which metrics matter the most to our venture. We’ll be sharing these regularly with others and comparing the actual results with predefined targets.
By creating these feedback loops we’ll start to narrow the long list of metrics we track down to a more focussed basket of metrics that capture the most important aspects of the business and clearly show how you are currently tracking. These are likely to be quite specific to our specific business model and stage, and will almost certainly change over time as old problems get solved and we grow into new areas with their own new problems.
In their book Lean Analytics, Alistair Croll and Benjamin Yoskovitz define useful metrics as those that have these five attributes:
To achieve metrics nirvana, we just need to choose one of these metrics to obsess about. And then do everything you can think of to improve it.
Focus in this situation means stripping our full list of metrics back to just those that indicate what we’re doing at the moment is working (or not!) - i.e. the things we can take action on.
Perhaps we’ve already identified the single metric that clearly correlates with our success.
Or maybe we’re still searching but know there is a specific area where we’re not doing as well as we think we can and want to really focus on that for a bit.
Often the best metric to choose will be a composite metric that captures several different and ideally competing aspects of the venture in a single number.
For example, in the early days of Trade Me we realised that one of the most impactful things we could to do drive our growth was increase the number of unique sellers listing items for sale. One seller listing lots of similar items was good, but lots of different sellers listing items was much better - more diverse listings meant more buyers, more feedback placed, more stories told by happy sellers, more word-of-mouth growth. We started tracking this metric daily and as a result were able to quickly identify a number of improvements we could make to increase this number - mostly by making the process of listing an item for sale much more obvious.
Another more recent SaaS example is Tomas Tunguz’s Cost of a Recurring Gross Profit Dollar metric (CRGPD). This is a great composite metric: it captures lots of different aspects of the SaaS business model in a single number (including the gross margin, acquisition cost and churn); it gives a single number which is immediately meaningful (if it’s costing more than $1 to gain $1 of recurring gross profit then that should immediately raise concern); and by tracking the number over time we can quickly see progress (a lower cost is better than a higher cost).
Action: Determine what is your most important metric at the moment, and then obsess about what you need do to improve that number?
They say that the hardest part of going for a run is putting on your shoes. Once we’ve done that then actually heading out the door is an obvious next step.
And it’s the same with all of this. Don’t let a blank spreadsheet scare us from taking the first step. Once we get started we’ll find that each of the levels are a natural progression from the last.
So get going and perhaps one day we’ll be one of those successful founders looking back talking about how much data helped us understand our venture and track our progress.
See also: Metrics That Matter: Ask Better Questions & Tell Better Stories.
Credit to Munjal Shah, the former founder and CEO of like.com, for this idea from a 2007 blog post that is sadly no longer online:
↩︎Launching a new site is like becoming the owner of a brand Russian nuclear power plant. You have a ton of dials with labels you can’t read. The only dial you can read is the amount of electricity (in our case revenue) and the temperature of the nuclear core (in our case number of clicks you are sending to merchants).
Obviously this doesn’t apply to any readers who already speak Russian. In that case, please substitute whatever language would be most indecipherable to you. ↩︎
In his book 21 Dog Years, Mike Daisey tells the story of the early customer support team at Amazon who were measured based on the average time taken to resolve a customer enquiry. They would hack this number by hanging up on customers who called with difficult questions. ↩︎
Say Something
Anywhere I would have followed you. But I’m giving up on you!
Feedback Loops
How can we be more explicit about the sort of feedback that is useful, right now.
The Mythical Startup
Can we update the fairytale version of how a technology startup becomes a success?
The Size of Your Truck
As we grow, take the time to understand unit economics.