Measuring Product KPIs and Metrics

Posted: 14 Jun 2023

Getting to product-market fit requires constantly iterating your product. It’s key to make sure that your changes are getting you closer, but measuring them is often tricky.

At early stages you won’t have enough data to measure improvements, whilst at later stages the data can be misleading. We’ve pulled together our latest PMF-Ready guide to help you navigate the world of product data and metrics.

“You can't improve what you don't measure.”

Peter Drucker

“Don't measure anything unless the data helps you make a better decision or change your actions.”

Seth Godin

The interplay between of metrics and strategy 🗺

It’s worth starting by being very clear about the different roles that strategy, goals, and metrics play. A good product company needs both strategy and metrics that work in harmony.

The Reforge Product Strategy Stack is a useful framework to give us some definitions:

  • Product strategy: the logical plan for how the product will help bring the company mission into being.
  • Product goals: the quarterly and day-to-day outcomes that measure progress against the strategy.

In other words, your product strategy creates a narrative for how your product will help the company win, while your product KPIs and metrics are measurements of progress along the journey.

So you need to lay your strategy out clearly before you can establish your metrics.

Metrics frameworks

Two main metrics frameworks are used by product companies: the North Star Framework and OKRs. Both are powerful but have their drawbacks. It’s useful to be aware of both of them, whilst understanding their limitations, and choosing the right path forward for your business.

North Star Metrics ⭐️

The North Star metric is one that aligns and guides the whole company.

Amplitude give us this definition: “The North Star Framework identifies a single, meaningful metric and a handful of contributing inputs. Product teams work to influence those inputs, which in turn drive the metric. The North Star is a leading indicator of sustainable growth and acts as a connective tissue between the product and the broader business.”

Amplitude have written a whole book on North Star Framework that is a useful read if you go down the path of using this methodology. The framework talks through how identify input metrics that ultimately live up to your

Growth consultant Ward van Gasteren has shared the North Star metrics of some well-known companies:

  • Uber: Rides per week.
  • WhatsApp: Messages sent.
  • Airbnb: Booked nights.
  • Spotify: Time spent listening.

While MRR or an equivalent might be tempting as a North Star metric, pre-PMF it can lead to over-optimising for extracting value from customers, rather than giving value to them.

The benefits of a North Star metric are obvious; it aligns the whole company and makes prioritisation very clear. The downside is linked; it’s sometimes an oversimplification to have one single objective as a company.

Reforge founder Brian Balfour shared the following graphic that shows how Spotify use their North Star metric to inform product work:

OKRs ⚙️

The other common framework for setting goals is “Objectives and Key Results” (OKRs). They were popularised in John Doerr’s book “Measure What Matters” and “Radical Focus” by Christina Wodtke. The latter is written almost as a novel, but both are seminal texts for OKRs in business.

Usage of OKRs varies between businesses, but generally they involve:

  • Setting company objectives that are big, audacious goals the company wants to achieve.
  • Assigning 3-5 key results to each objective, to know that you’re making progress against the goals.
  • Repeat for departments and/or teams, and eventually individuals.

As with North Star metrics, the benefits are pretty clear. OKRs force everyone to align with common goals for the whole company, and make it much easier to work on only the highest priority projects.

They were used with great success at Google, which launched them into becoming something of a staple for modern product-led companies.

Drawbacks of these frameworks

There are unfortunately some downsides to using these frameworks before having product-market fit:

  • It’s hard to set clear goals pre-PMF: There is so much uncertainty around who your target customers should be, their problems, and your ability to solve for them, that setting concrete goals can be hard.
  • Your focus might need to change frequently: Finding PMF is fast-paced and uncertain. Frameworks are fantastic for helping companies to focus, which is why the typical cadence for OKRs is quarterly. But pre-PMF companies might need to shift priorities and focus on a monthly basis, or even more frequently. These frameworks can get in the way of quick pivots.
  • It takes time to get them right: Experienced operators might be able to adopt OKRs and get use them successfully and quickly. But others will take multiple quarters to see value. That is time that startups simply don’t have.

Figma CPO Yuhki Yamashita got Figma to stop using OKRs when he joined. He opted instead for “headlines”: a claim a team could make by a certain point. For example, a team could say “Figma is the most efficient way to design”, and then offer qualitative and quantitative ways to prove the claim.

This helped Yamashita to clearly see a team’s big picture vision, and measure progress towards that. He says “It recognized the reality that some things, like the core experience of Figma, *are* hard to measure and can’t be reduced to a single metric”.

They have since gone back to OKRs now that they have someone leading on Data Science.

Andrew Chen of Andreessen Horowitz (and formerly in growth at Uber) writes about how OKRs aren’t necessarily appropriate for pre-PMF startups – again making the point that at this uncertain stage, companies need to be extremely agile, whereas OKRs prevent focus from changing too quickly.

Our advice is to take the lessons from these frameworks but don’t get caught up in the dogma. You should have few priorities, define success upfront, and measure progress as objectively as you can.

A rule-of-thumb framework to use could be:

  • Cadence: Set a cadence that makes the most sense for your team.
  • Mission: Choose something that you’re going to try to achieve.
  • Metric: Choose a metric that will indicate success.
  • Reporting: Frequently share progress against those metrics.

This is clearly similar to OKRs, but it’s far looser as a general framework and will give teams a lot of the benefits without adding too much process.

What makes a good product metric ✅

Every product is unique and their metrics may need to be too. This might mean coming up with your own original measures and evaluating them. A good product metric is:

One the team can move: A product team needs to be able to have an impact on their metric.

One the team can measure: They need to be able to capture the metric and analyse it.

One with quick feedback: They need to be able to tell quickly if their product work is having an impact. Waiting to see the impact on 90-day retention is too long.

One that you care about: It needs to make a difference if these metrics are actually hit. Email signups to a mailing list don’t really impact the business.

One they can isolate their impact on: A metric like customer retention is dependent on so many factors that it’s very hard for a team to measure the impact of their work.

💡 Balancing leading + lagging: proxy metrics

Finding the right balance between leading and lagging metrics is hard. Lagging metrics typically tell you more about what you actually care about as a business (e.g. retention and revenue), but it can take a long time to measure change.

One way around this is to use proxy metrics for shorter-term work. Gibson Biddle, former VP Product at Netflix, defines these as a “stand-in for your high-level metric”. Basically it’s when you use a metric to prove something else.

For example, if you noticed that customers that used your native companion app were more likely to be retained after 6 months, you might use mobile app downloads as a proxy metric. You don’t want to focus on it long-term, as download numbers aren’t success in and of itself. But if there’s a clear link to the metric you do care about, it can stand in as a useful proxy.

Tips for specific product metrics 💡

1/ Retention:

  • Retention is a measure of all the value a customer gets from your product. It’s affected by so many things (including price) that it’s hard for a product team to consistently impact it. It’s like asking a hospital to increase life expectancy in the area around it. It’s a great measure for certain projects, but hard for one team to own the metric long-term.
  • When measuring retention, make sure to measure cohort retention. Cohorts are groups of users that join at the same time. Analysing cohorts separately lets you filter out brand new users from existing ones, for more accurate retention data.
  • Retention is a lagging indicator. You won’t find PMF by pleading customers not to churn at the last minute. Instead, improve retention by finding ways to create so much value that they’d never consider leaving in the first place.

2/ Engagement:

  • Engagement obviously means different things to different types of companies. Think about your users and how they get value from your product. As an example, Netflix users get value from watching content not browsing the site.
  • In his talk “The Only Product Metric That Matters”, John Elman talks about the importance of measuring usage and engagement. His advice is to define the core action that means users get value from your product and the frequency that you think they should be getting value from it. Then ask yourself: “How many times do our users perform the core action in the expected cycle?”
  • Don’t get too hung up on the length of the expected cycle – you can change it over time. Apparently Facebook started out measuring monthly active users, then weekly, then daily as they improved the product’s stickiness.

3/ Activation:

  • Similarly to the above, choose a point at which you think new users will discover value in your product. Typically speaking, you want to get them there as frequently and quickly as possible.
  • Don’t worry about finding the exact perfect metric. Facebook famously tried to get every user to connect with 7 friends, within the first 10 days. This became a rallying cry for the whole company. There’s no magic change at these numbers – 7 friends in 11 days wouldn’t be much different. But just having a memorable activation metric that they believed in was good enough to align the whole company.

4/ Conversion:

  • Conversion can be impacted by many different factors, so running A/B tests to measure change is very important.
  • A team at Wise once shipped an improvement to the money transfer flow aiming to improve conversion. But when they checked their conversion metric, they saw it had fallen! There was a lot of confusion until they dug into conversion by channel during their test. Conversion had improved for every channel! But the marketing team had adjusted spend, resulting in greater mix of lower-intent traffic, which led to overall conversion reducing.
  • Because of this, it can be hard for teams to target specific conversion rates for their work (e.g. we want to hit 10% signup-to-payment conversion). Instead, it’s better for them to measure the impact of the work they did, saying “we want to improve conversion rates X% compared to doing nothing”.

Anchor metrics ⚓️

Goodhart’s law states that as soon as a measure becomes a target, it stops being a good measure. This is because people game the metric (subconsciously or otherwise) which leads to unintended outcomes.

Despite the best of intentions, it will always be tempting for a team to over-optimise for their target metric. To counteract this, it’s useful to have an anchor metric that ensures they’re not harming something else.

For example, if a team had the metric of reducing customer support contacts, they might do that by making customer support harder to find. That would improve the metric but not in a way that really helps the company. By having an anchor metric like “do no harm to customer satisfaction” the team can ensure that they act in the correct manner.

This is also why it’s best to not use ratios for product metrics (e.g. avoid “increase the % of users that pay for premium features”). This metric can be improved by either increasing the amount of paying users, or decreasing the amount of non-paying users. Better to have absolute metrics (e.g. “increase the amount of users that pay for premium features”).

Some common metrics mistakes 🙈

Setting metrics can be hard! We’ve listed some of the more frequent mistakes that we see founders make, to help avoid them.

Hard to remember: Don’t choose metrics that are confusing, or have so many that they’re hard for people to recall. If you want your team to make decisions with a metric in mind, it’s critical that they can remember them. Have as few as possible, and make sure they’re simple enough for everyone to understand.

Ship goals: Sometimes product teams will have a measure of success like “get feature X live to customers on time”. This is sometimes necessary, but when repeated, it leads to teams optimising for output rather than outcomes. Have metrics instead that show that the feature is being used and adding value to your customers.

Setting and forgetting: It takes time to get value from the set of metrics that you choose. Often teams will go through the process of choosing their metrics and targets, and then not report on them. This destroys any value you’ll get from them. Make sure that you report on your chosen metrics and reflect on progress. Without this it’s very hard to create accountability.

Don’t cargo-cult: In programming, “cargo-culting” is when you of copy code from one program into another where it doesn’t serve any real purpose. It’s tempting to choose a key metric that made sense for Facebook or Spotify, and copy that. Avoid this – your business is unique and your measures of success probably should be too.

But… an ok metric is better than no metric: Often people argue about which metric is the perfect one, and end up with no measures. Having a metric that somewhat reflects progress is much, much, much better than having no measures at all. So use this as a guide, but don’t get hung up on dogma.

💡 Author’s opinion: Targets

I personally don’t like setting targets for product teams. Targets are often set arbitrarily, with no real tangible difference if they’re hit by a certain deadline or not. Over time people start to realise this and it makes targets feel illegitimate.

While well-set targets can encourage better performance, the wrong target may encourage under-performance; either through being unrealistically high, or by setting a benchmark too low.

If you do want to set targets, let the team set their own. This keeps them legitimate to the team and creates real accountability.

What if you don’t have much data? 🔮

In the earliest stage of a startup’s life, you won’t have much data to measure. In these times your best source of data will be frequently speaking to your users (and potential users). You’ll be relying on quantitative data, so do this as much as you can.

As things progress and you get more users, you’ll get more data. You can start to measure it, but be wary of “ticker watching”. Ticker watching is when you watch a metric that naturally varies each week and overreact to the movements.

It’s a risk when your data sets are small. When you have 10 customers, 10% churn might be normal as one customer leaves. When you have 10,000 customers, you’re unlikely to get 10% churn in a week. Make sure to bear this in mind in the early days.

As you grow, you may be able to run experiments as ways of measuring improvements. More on that below.

A quick guide to A/B tests 🔬

When you introduce a new feature, you want to be sure that you are making your product better. Otherwise you run the risk of having a bloated product that is packed with features but hard to use.

It’s hard to know that you’ve improved the product without a scientific test. It’s rare to eyeball a graph and be able to pinpoint the change to your metric. Sometimes metrics change because of seasonality, or a change by another team. Sometimes they just move randomly as standard deviation.

A/B tests, or “split tests” are a way to measure the improvement from a new feature. They use the scientific method to get certainty on the impact of a feature. The method sets out to prove that an observed change in the metric is at least 95% likely due to the new feature, rather than just random chance.

Experimentation is a broad topic that deserves a whole post on it’s own, but here’s a quick run-down:

  • Split users into two groups: one group (test) sees the new feature and the other (control) doesn’t.
  • Choose a target variable that will be your measure of success (e.g. conversion to paid customer).
  • Work out how long you’ll need to run the test for to get 95% confidence, using a P-value calculator.
  • Run the test for that period of time.
  • Analyse the results. If the feature improves things, ship it! If not, try to understand why not and make a call on the next steps.

Experimentation is important, because while you’re finding PMF many of your ideas will be wrong. The reality of great products is a lot of stuff that just never worked out. Testing and learning is part of that journey.

How to use metrics but retain your soul 👻

Founders are often wary of relying on data too much, and losing the “soul” of their product. If you focus purely on data, your product will start to reflect that. You run the risk of shipping every feature that moves a metric and losing the bigger picture about what is good for your users.

Facebook’s homepage is a good example – it looks like every element has been optimised for engagement, to the detriment of wider usability. I don’t have access to Facebook’s data so I don’t know for sure, but that’s what it looks like to me

So – how do you use data but avoid building a Frankenproduct?


  • Product principals: Keep a list of what’s important in your product. What do you care about, even if the data says otherwise? This should guide you, to make sure that you’re not over-optimising for whatever gets the most short-term engagement.
  • Keep talking to your customers: Speaking to real people will give you insight that data on it’s own won’t. Make sure to keep those qualitative conversations going even when you get more data. Use focus groups and usability tests before releasing features, to understand what is happening behind the data.
  • Keep placing big bets: Using data to drive your product can lead people to focus on projects that are low-risk and easy to measure. Dave Thomson of Skyscanner wrote about how chasing quick wins can quickly be detrimental to your business.

📚 Further reading