Use these simple steps to improve how we measure and report our progress.
Start simply by asking what are the most important facts in your organisation and how many people know them
— Hans Rosling, Factfulness
Successful founders, looking back, often talk about how they used data to track their progress and inform their decisions.
But how did they get started?
During the early days of a new venture, it’s far more common for both founders and investors to feel completely overwhelmed or outright misled by data than to feel informed or driven by it. Even when we understand the importance of looking at our numbers constantly and sending regular updates, starting with a completely blank spreadsheet is still a daunting prospect.
So please don’t panic if you feel like that. It’s completely normal.
Using data like this is a behaviour that we can learn.
In the very beginning all we have are numbers.
The best analogy I have is the science fiction nightmare of waking up to find we’re locked in the control room of an abandoned Russian nuclear power plant that is just about to melt down.1 We can see the control panel, with all the dials and switches, and we know we need to do something, but none of the labels make any sense.
This obviously doesn’t apply to any readers who already speak Russian and so would cope better in this scenario. Please substitute whatever language would be most indecipherable to you!
All we can do is rapidly learn which control is which by trial and error. It’s impossible to know exactly which of the numbers we have are going to matter most to our venture as we grow, so to get started the best we can do is track everything, beginning with a standard template of metrics that are applicable to our business model.
Actions:
It’s tempting at this point to jump straight into analysis.
But wait!
Often it’s not the complexity of the maths we do or how many three-letter-acronyms (TLAs) we have that makes the difference - it’s the consistency of the habits that are formed early around regularly updating our numbers, looking at the trends and sharing them with somebody who can help us decipher them. Maybe we can do that all by ourselves, but more likely it will be a team member, an external investor or an advisor. It’s harder than we ever realise to punch holes in our own reality distortion field.
It’s shocking how many founders never get beyond this step. Despite starting with the best of intentions, gradually the frequency of updates starts to dwindle. The best sign I have that a startup I’ve invested in is dead, or near death, is the silence.
Why does this happen?
Sometimes it’s because our numbers are not in a format that is easily shared. More commonly, we’re embarrassed because the results aren’t what we’d hoped for and confidently predicted they would be. That’s understandable. The natural inclination is to wait until the numbers are better. Maybe we can fix things before anybody notices? But when we do that, we keep information out of the hands of the very people who can help the most. As soon as we know, or even as soon as we have a sense that things are off-track, we need to say that out loud. So often problems are much more easily fixed than we fear when we ask for help, especially when we catch them early enough.
Preparing updates doesn’t have to be complex or time consuming.
We can start by sending out the most basic financial details: how many new customers we have, how much we earned and spent in the last month and how much cash we have left in the bank.
Later, as the business grows, we can include some commentary about the current constraints. Start with the question: why aren’t we growing faster? Talk about people: new hires, open roles, how we’re feeling about the internal culture. Highlight the key numbers: how much are we spending to acquire each new customer, how much does it cost to support each customer each month, and how many customers cancel (or “churn”) each month? Ask others in the team to contribute a short paragraph about their area of the business.
These are all details we should have at our fingertips now (see Level 1 above). The details we share with investors and advisors should be a subset of our management reports.
The goal is: simple updates on a regular cadence.
The most important thing is to be consistent and regular rather than amazing infrequently but neglectful the rest of the time. When we regularly review our numbers with somebody who is objective it makes a big difference, and once the habit is established it becomes something that we do intuitively.
An even bigger payback from regularly sharing results with investors comes when we want to raise additional capital in the future. There is nothing worse for an investor than an email from out of the blue asking for more money. If we’ve taken the time to keep everybody informed then we’ll spend much less time telling them where we’ve been, so we can focus on where we want to go next, and we’ll likely find investors are much more enthusiastic about continuing to be part of that journey.
Even when we think we have nothing interesting to say, say something.
Actions:
Once we have a handle on the basic metrics associated with our business, and we’ve formed the habit of updating and sharing them regularly, the next job is to be more savvy about our analysis.
Often the most interesting metrics are not the things that we can record directly - e.g. the number of unique visitors to our website, the number of paying customers we have, or the revenue we earned - but the combinations, derivatives and countermeasures.
What are the ratios that measure our progress?
These are more nuanced than the raw numbers, because they are more easily compared across different time periods and to industry benchmarks.
For example, if we have an online retail business, rather than looking at the number of new customers we gained or how much we spent on advertising, combine those to calculate our cost of acquisition (often abbreviated to CAC) - that is, how much we spent on average over the last period to acquire each new customer.
Or, for any kind of business that is not yet profitable, rather than recording how much cash we have in the bank, and how much cash we are burning (total expenses net of revenue), combine those to calculate our runway - i.e how many months we can continue in that mode before we run out of money?
What are the first or second derivatives of our basic metrics?
For example, if we have a SaaS business, in addition to measuring our average revenue per subscriber (often abbreviated to ARPU), calculate the change - is the ARPU increasing or decreasing?
Often, to get value from derivatives we need to look back over a few periods, so we can really understand the trend and we are not distracted by one abnormal data point.
Whenever we have a positive metric, it’s useful to ask: what is the countermeasure?
Often when we focus on one thing, such as growing the number of customers we have, we can lose sight of the collateral damage this is causing in other areas of the business. Having a good countermeasure will ensure that we have healthy growth.
A good way to identify countermeasures is to put on our tinfoil hat and ask: if somebody in the team was maliciously trying to juice this metric, how would we tell?
For example, if we have a marketplace business, then one of our key metrics is the inventory we have - i.e. the number of active listings on a classified site. It’s not just the raw amount of inventory that matters, but also the quality. We can also measure the ratio of sales to inventory (often called the sell-through rate), or the ratio of unique sellers to inventory (the listings per unique seller), as a way of checking that we have valuable listings and not just lots of them.
Or, a classic example, if we measure the average time it takes to respond to a customer enquiry then, as a countermeasure, we should also look at the quality of the responses, so that our customer service team don’t just send poorly-considered replies, but actually try to solve the problem the customer is experiencing.2
Actions:
Once we’re collecting, analysing and sharing our metrics, the next step is to create feedback loops, so that we can constantly improve.
Choose the metrics we think are most important and set a target for each of these - either a level (e.g. more than 100 new customers this month) or a range (e.g. average sales per day of between $500 and $700). These could be driven by the business model, the cash position or just arbitrary growth objectives. Either way, we should try to be realistic about how quickly and how impressively we can make changes and achieve these results.
Once our goals are defined, whenever we update our metrics we can compare our actual performance to where we imagined we would be, and think about the gaps.
This is a form of debrief, and if we do it regularly we’ll get much better at it.
Remember, the questions to ask are: what went well and what didn’t, what surprised us, and what did others do in response?
We should write these things down as we review our metrics and consider why we did or didn’t hit our targets. We can look back at those in the future and ask if those reasons have stood the test of time, are recurring excuses (which is a problem!) or have been something that we were able to subsequently correct.
Most of the things that are easily measured are lagging indicators, showing what has happened in the relatively recent past.
It’s much more valuable if we can uncover a leading indicator - i.e. something that helps us to know that we’re already on the right track.
Possibly the most famous example of a leading indicator is the engagement measure used at Facebook. They discovered that if a new user has connected with seven friends within 10 days of creating an account they typically remain much more engaged as a user. That’s a hugely useful thing to understand, because the team can then focus on on how to get as many new users as possible across that threshold.
Actions:
By now we should have a much better grip on which metrics matter the most to our venture. We’ll be sharing these regularly with others and comparing the actual results with predefined targets.
By creating these feedback loops we narrow the long list of metrics we track down to a more focussed basket that captures the most important aspects of the business and clearly shows how we are tracking. These are likely to be quite specific to our business model and stage, and will almost certainly change over time as old problems get solved and we grow into new areas with their own new problems.
We need to strip our full list of metrics back to just those that indicate if what we’re doing at the moment is working (or not) - i.e. the things we can take action on.
Choose one of these metrics to obsess about, then do everything we can think of to improve it.
Perhaps we’ve already identified the single metric that clearly correlates with our success. Or maybe we’re still searching but just want to focus on a specific area where we’re not doing as well as we could.
Often the best metric to choose will be a composite metric that captures several different and ideally competing aspects of the venture in a single number.
At Trade Me, in the early days, this was the number of unique sellers listing items for sale. We correctly guessed that if we could increase that then it would create a positive feedback loop: more sellers would mean more diverse items for sale, more bids, more successful auctions, more feedback placed and critically more word-of-mouth growth due to people telling their family and friends that they’d be crazy not to list their unused stuff for sale too. We started tracking this metric daily and as a result were able to quickly identify a number of improvements we could make to increase this number - mostly by making the process of listing an item for sale much more obvious.
Another more recent SaaS example which became a key part of both Vend and Timely board packs when I was working with those companies is Tomas Tunguz’s Cost of a Recurring Gross Profit Dollar metric (CRGPD).3 This is a great composite metric: it captures lots of different aspects of the SaaS business model in a single number (including the gross margin, acquisition cost and churn); it’s immediately meaningful - if it’s costing more than $1 to gain $1 of recurring gross profit that should straight away raise concern; and by tracking the number over time we can quickly see progress - a lower cost is better than a higher cost.
Actions:
We need to avoid vanity metrics.
These are the metrics that make us feel good - the number of visitors to our website, the number of likes on our social media posts, or the number of times we’re mentioned in the media.
They favourably compare our performance to others, or to arbitrary benchmarks. They tell a positive story.
For those of us who are competitive by nature, the comparisons baked into vanity metrics can be very motivating. But we need to be careful we don’t fall into the trap of comparing our insides with others outsides.
For example, I found it confusing when people celebrated how quickly the team at Vend was growing. Knowing when headcount tips from being a vanity metric to being vital is one of the keys to successfully scaling a venture beyond the startup stage. But unless we know the revenue per employee a bigger team just means a bigger payroll.
We also need to be wary of vanity milestones.4
These are the things that make for good press releases and generate attention. They give us something to publicly celebrate.
For example, winning awards, partnerships with much bigger companies (which rarely, if ever, translate into significant sales for the smaller partner) or attracting small investment amounts from celebrity investors.
We need to remember causation vs. correlation. Lots of successful companies win business awards. But winning business awards is only loosely related to the things we need to do, have and be in order to build a successful business.
Vanity metrics and milestones are invigorating but also flattering and misleading. Our metrics shouldn’t be results we report only when they are great, in order to get accolades. They should be the indicators that we use to determine what is and (often more importantly) isn’t currently working.
What’s the alternative?
We really need clarity metrics.5
These highlight trends in our performance and ideally help us decide what to do next. They are actionable and fundamental.6
For example, average usage times per user, average time spent on-boarding, churn/retention rates by cohort, revenue:acquisition cost ratios, etc.
Clarity metrics are humbling and sometimes exhausting.
It’s easy to be dismissive about vanity metrics and milestones, but…
The average finishing time for a marathon on Strava is 3:58.25 - i.e. just under 4 hours. The technical difference between a 4:01 marathon and a 3:59 marathon is insignificant - just 2.8 seconds per kilometre (perhaps a few strides). But the psychological difference is huge. So let’s not pretend that vanity metrics don’t have the potential to positively motivate performance.
In addition to the metrics we use to inform our actions, it’s useful to have some hero metrics to demonstrate progress to people outside of the company, and to generate buzz that might tempt some of them to get more involved.
The best example I’ve seen of this recently is an environmental restoration project in the Coromandel, where the metric they talk about is “the number of complaints about bird song noise”. The people running the project figured that was a great way to show they were having an impact, and even more, to signal to everybody in the area what the real purpose of the project is - i.e. it’s not just about planting trees and clearing traps.
We just need to always keep in mind which metrics are useful because they make us (and our team) feel good and which are useful because they help us decide what to do next.
Those are likely different metrics.
They say that the hardest part of going for a run is putting on running shoes. Once we’ve done that then heading out the door is an obvious next step.
And it’s the same with metrics. We can’t let a blank spreadsheet scare us from taking the first step. Once we get started we’ll find that each of the levels are a natural progression from the last.
So get going and perhaps one day we’ll be a successful founder looking back talking about how much data helped us understand our venture and track our progress.
Credit to Munjal Shah, the former founder and CEO of like.com, for this idea from a deleted 2007 blog post:
↩︎Launching a new site is like becoming the owner of a brand Russian nuclear power plant. You have a ton of dials with labels you can’t read. The only dial you can read is the amount of electricity (in our case revenue) and the temperature of the nuclear core (in our case number of clicks you are sending to merchants).
In his book 21 Dog Years, Mike Daisey tells the story of the early customer support team at Amazon who were measured based purely on the average time taken to resolve a customer enquiry. They would hack this metric by hanging up on customers who called with difficult questions. ↩︎
A New Way to Calculate a SaaS Company’s Efficiency by Tomas Tunguz, 1st December 2018. ↩︎
Vanity milestones by Chris Dixon, 11th September 2012. ↩︎
These names are not mine. Eric Ries talked about vanity vs. actionable metrics in The Lean Startup. First Round Capital also published a good breakdown of vanity vs. clarity metrics, with specific examples for different business models. ↩︎
In Lean Analytics, Alistair Croll and Benjamin Yoskovitz define useful metrics as those that have these five attributes:
Machines & Phases
There are so many different ways to measure a startup. It’s easy to drown in metrics. How do we separate the signal from the noise?
Feedback Loops
How can we be more explicit about the sort of feedback that is useful, right now.
The Size of Your Truck
As we grow, take the time to understand unit economics.
Anything vs Everything
How do we choose what to focus on, and have the conviction to say “no” to everything else?
Flailing
Here is some unusual advice for people working on a startup, or thinking about it: swim.
How to Be Wrong
There are only three ways to be wrong about our impact: neglect, error and malice.