Getting to product-market fit requires constantly iterating your product. It’s key to make sure that your changes are getting you closer, but measuring them is often tricky.
At early stages you won’t have enough data to measure improvements, whilst at later stages the data can be misleading. We’ve pulled together our latest PMF-Ready guide to help you navigate the world of product data and metrics.
“You can't improve what you don't measure.”
Peter Drucker
“Don't measure anything unless the data helps you make a better decision or change your actions.”
Seth Godin
It’s worth starting by being very clear about the different roles that strategy, goals, and metrics play. A good product company needs both strategy and metrics that work in harmony.
The Reforge Product Strategy Stack is a useful framework to give us some definitions:
In other words, your product strategy creates a narrative for how your product will help the company win, while your product KPIs and metrics are measurements of progress along the journey.
So you need to lay your strategy out clearly before you can establish your metrics.
Two main metrics frameworks are used by product companies: the North Star Framework and OKRs. Both are powerful but have their drawbacks. It’s useful to be aware of both of them, whilst understanding their limitations, and choosing the right path forward for your business.
The North Star metric is one that aligns and guides the whole company.
Amplitude give us this definition: “The North Star Framework identifies a single, meaningful metric and a handful of contributing inputs. Product teams work to influence those inputs, which in turn drive the metric. The North Star is a leading indicator of sustainable growth and acts as a connective tissue between the product and the broader business.”
Amplitude have written a whole book on North Star Framework that is a useful read if you go down the path of using this methodology. The framework talks through how identify input metrics that ultimately live up to your
Growth consultant Ward van Gasteren has shared the North Star metrics of some well-known companies:
While MRR or an equivalent might be tempting as a North Star metric, pre-PMF it can lead to over-optimising for extracting value from customers, rather than giving value to them.
The benefits of a North Star metric are obvious; it aligns the whole company and makes prioritisation very clear. The downside is linked; it’s sometimes an oversimplification to have one single objective as a company.
Reforge founder Brian Balfour shared the following graphic that shows how Spotify use their North Star metric to inform product work:
The other common framework for setting goals is “Objectives and Key Results” (OKRs). They were popularised in John Doerr’s book “Measure What Matters” and “Radical Focus” by Christina Wodtke. The latter is written almost as a novel, but both are seminal texts for OKRs in business.
Usage of OKRs varies between businesses, but generally they involve:
As with North Star metrics, the benefits are pretty clear. OKRs force everyone to align with common goals for the whole company, and make it much easier to work on only the highest priority projects.
They were used with great success at Google, which launched them into becoming something of a staple for modern product-led companies.
There are unfortunately some downsides to using these frameworks before having product-market fit:
Figma CPO Yuhki Yamashita got Figma to stop using OKRs when he joined. He opted instead for “headlines”: a claim a team could make by a certain point. For example, a team could say “Figma is the most efficient way to design”, and then offer qualitative and quantitative ways to prove the claim.
This helped Yamashita to clearly see a team’s big picture vision, and measure progress towards that. He says “It recognized the reality that some things, like the core experience of Figma, *are* hard to measure and can’t be reduced to a single metric”.
They have since gone back to OKRs now that they have someone leading on Data Science.
Andrew Chen of Andreessen Horowitz (and formerly in growth at Uber) writes about how OKRs aren’t necessarily appropriate for pre-PMF startups – again making the point that at this uncertain stage, companies need to be extremely agile, whereas OKRs prevent focus from changing too quickly.
Our advice is to take the lessons from these frameworks but don’t get caught up in the dogma. You should have few priorities, define success upfront, and measure progress as objectively as you can.
A rule-of-thumb framework to use could be:
This is clearly similar to OKRs, but it’s far looser as a general framework and will give teams a lot of the benefits without adding too much process.
Every product is unique and their metrics may need to be too. This might mean coming up with your own original measures and evaluating them. A good product metric is:
One the team can move: A product team needs to be able to have an impact on their metric.
One the team can measure: They need to be able to capture the metric and analyse it.
One with quick feedback: They need to be able to tell quickly if their product work is having an impact. Waiting to see the impact on 90-day retention is too long.
One that you care about: It needs to make a difference if these metrics are actually hit. Email signups to a mailing list don’t really impact the business.
One they can isolate their impact on: A metric like customer retention is dependent on so many factors that it’s very hard for a team to measure the impact of their work.
Finding the right balance between leading and lagging metrics is hard. Lagging metrics typically tell you more about what you actually care about as a business (e.g. retention and revenue), but it can take a long time to measure change.
One way around this is to use proxy metrics for shorter-term work. Gibson Biddle, former VP Product at Netflix, defines these as a “stand-in for your high-level metric”. Basically it’s when you use a metric to prove something else.
For example, if you noticed that customers that used your native companion app were more likely to be retained after 6 months, you might use mobile app downloads as a proxy metric. You don’t want to focus on it long-term, as download numbers aren’t success in and of itself. But if there’s a clear link to the metric you do care about, it can stand in as a useful proxy.
1/ Retention:
2/ Engagement:
3/ Activation:
4/ Conversion:
Goodhart’s law states that as soon as a measure becomes a target, it stops being a good measure. This is because people game the metric (subconsciously or otherwise) which leads to unintended outcomes.
Despite the best of intentions, it will always be tempting for a team to over-optimise for their target metric. To counteract this, it’s useful to have an anchor metric that ensures they’re not harming something else.
For example, if a team had the metric of reducing customer support contacts, they might do that by making customer support harder to find. That would improve the metric but not in a way that really helps the company. By having an anchor metric like “do no harm to customer satisfaction” the team can ensure that they act in the correct manner.
This is also why it’s best to not use ratios for product metrics (e.g. avoid “increase the % of users that pay for premium features”). This metric can be improved by either increasing the amount of paying users, or decreasing the amount of non-paying users. Better to have absolute metrics (e.g. “increase the amount of users that pay for premium features”).
Setting metrics can be hard! We’ve listed some of the more frequent mistakes that we see founders make, to help avoid them.
Hard to remember: Don’t choose metrics that are confusing, or have so many that they’re hard for people to recall. If you want your team to make decisions with a metric in mind, it’s critical that they can remember them. Have as few as possible, and make sure they’re simple enough for everyone to understand.
Ship goals: Sometimes product teams will have a measure of success like “get feature X live to customers on time”. This is sometimes necessary, but when repeated, it leads to teams optimising for output rather than outcomes. Have metrics instead that show that the feature is being used and adding value to your customers.
Setting and forgetting: It takes time to get value from the set of metrics that you choose. Often teams will go through the process of choosing their metrics and targets, and then not report on them. This destroys any value you’ll get from them. Make sure that you report on your chosen metrics and reflect on progress. Without this it’s very hard to create accountability.
Don’t cargo-cult: In programming, “cargo-culting” is when you of copy code from one program into another where it doesn’t serve any real purpose. It’s tempting to choose a key metric that made sense for Facebook or Spotify, and copy that. Avoid this – your business is unique and your measures of success probably should be too.
But… an ok metric is better than no metric: Often people argue about which metric is the perfect one, and end up with no measures. Having a metric that somewhat reflects progress is much, much, much better than having no measures at all. So use this as a guide, but don’t get hung up on dogma.
I personally don’t like setting targets for product teams. Targets are often set arbitrarily, with no real tangible difference if they’re hit by a certain deadline or not. Over time people start to realise this and it makes targets feel illegitimate.
While well-set targets can encourage better performance, the wrong target may encourage under-performance; either through being unrealistically high, or by setting a benchmark too low.
If you do want to set targets, let the team set their own. This keeps them legitimate to the team and creates real accountability.
In the earliest stage of a startup’s life, you won’t have much data to measure. In these times your best source of data will be frequently speaking to your users (and potential users). You’ll be relying on quantitative data, so do this as much as you can.
As things progress and you get more users, you’ll get more data. You can start to measure it, but be wary of “ticker watching”. Ticker watching is when you watch a metric that naturally varies each week and overreact to the movements.
It’s a risk when your data sets are small. When you have 10 customers, 10% churn might be normal as one customer leaves. When you have 10,000 customers, you’re unlikely to get 10% churn in a week. Make sure to bear this in mind in the early days.
As you grow, you may be able to run experiments as ways of measuring improvements. More on that below.
When you introduce a new feature, you want to be sure that you are making your product better. Otherwise you run the risk of having a bloated product that is packed with features but hard to use.
It’s hard to know that you’ve improved the product without a scientific test. It’s rare to eyeball a graph and be able to pinpoint the change to your metric. Sometimes metrics change because of seasonality, or a change by another team. Sometimes they just move randomly as standard deviation.
A/B tests, or “split tests” are a way to measure the improvement from a new feature. They use the scientific method to get certainty on the impact of a feature. The method sets out to prove that an observed change in the metric is at least 95% likely due to the new feature, rather than just random chance.
Experimentation is a broad topic that deserves a whole post on it’s own, but here’s a quick run-down:
Experimentation is important, because while you’re finding PMF many of your ideas will be wrong. The reality of great products is a lot of stuff that just never worked out. Testing and learning is part of that journey.
Founders are often wary of relying on data too much, and losing the “soul” of their product. If you focus purely on data, your product will start to reflect that. You run the risk of shipping every feature that moves a metric and losing the bigger picture about what is good for your users.
Facebook’s homepage is a good example – it looks like every element has been optimised for engagement, to the detriment of wider usability. I don’t have access to Facebook’s data so I don’t know for sure, but that’s what it looks like to me
So – how do you use data but avoid building a Frankenproduct?
📚 Further reading
