Success metrics: how to set them before you build

What are success metrics, and why set them before building?

Success metrics are the numbers that tell you whether a feature produced the change in user behaviour the team expected. Agree them before building: one primary metric and one or two guardrail metrics, each with a baseline, a target, a time window and a data source.

Here is an illustrative situation. You are the PO at Bếp Xinh, a fictional online kitchenware shop. In backlog refinement, the team discusses a feature that pre-fills a returning customer's saved address at checkout. The sales lead says: “Once this ships, conversion will go up.” A developer asks: “By how much, for it to count as a success?” Nobody answers. You write in the ticket: “Goal: increase conversion.”

Six weeks after launch, the review turns into an argument. Sales looks at monthly revenue, which is up thanks to a promotion. Marketing looks at site visitors, which are down. You look at the order rate, which has crept up slightly, but you cannot say whether that is good enough. Everyone picks a number, and every number sounds reasonable.

This article walks through one way to agree metrics before the team starts building, so that the post-launch review is just a matter of opening one page and comparing numbers. Reading results after launch is covered in the feature measurement guide; this one focuses on the preparation.

• Measuring a feature after launch

Why is it hard to agree success metrics?

Everyone agrees that measuring matters. The difficulty is elsewhere.

• The goalposts move after the results arrive. Without an agreed number, it is easy to pick whichever metric looks best after launch, and any result can be told as a success.
• People mean different things by “conversion”. Some divide by site visitors, others by people who started checkout. Same word, two different numbers.
• There is no baseline. If you do not know the number before the change, you cannot say the change made it better.
• The data does not exist yet. If the event you need to count is not tracked, tracking has to go into scope. Finding that out after launch is too late.
• Committing to a number feels risky. A target can sound like a promise. But without a number, there is no way to learn anything from the result.

Agreeing early also aligns the team. When developers, designers and sales all watch the same number, small decisions during the build, such as removing a step or adding a field, share one yardstick.

How do you set success metrics for a feature?

This article suggests a simple chain: the problem, then the expected behaviour change, then the metrics, then the baseline and target, and finally the data source and decision rules. Each link answers one question and depends on the one before it.

Step 1: write the problem and the expected behaviour as two sentences. Problem: “Returning customers on mobile abandon checkout at the address step.” Expected behaviour: “With this feature, returning customers will get through the address step faster and abandon it less.” The second sentence is about people, not screens. If you cannot write it, the problem is still vague, and a few user interviews may help.

Step 2: choose one primary metric. It answers the behaviour sentence directly, and it is the only number used to say whether the feature worked. Then choose one or two guardrail metrics: things that could get worse while the team pushes the primary number up. Microsoft Research describes guardrails as aspects of the product the team does not want to degrade, even though they will not necessarily improve, with examples such as page load time and error rates. The post-launch guide suggests two or three measures; here those two or three get distinct roles: one primary metric decides success, and the rest are guardrails or lagging signals to follow.

Step 3: record the baseline, the target and the time window. Take the baseline from a recent period long enough that one unusual day does not distort it. The target is a specific number, not “up”. The time window is the measurement period plus a review date. If few users fall inside that window, a difference of a few points may be ordinary noise. When unsure, ask whoever works with the data before you commit. If there is no baseline because the event is not tracked yet, you have two options: track it for a few weeks before launch to get one, or roll the feature out gradually to part of your users and compare them with the group that does not have it yet over the same period.

Step 4: record the data source and the decision rules. The source says which event the number is counted from, in which tool or table, and who owns it. The decision rules are answers written in advance for each outcome: what you do if it hits the target, if it improves but falls short, if it barely moves or drops, or if a guardrail crosses its limit. Writing them before you see results is the simplest way to stop the goalposts moving.

Researchers at Google also proposed the HEART framework together with a process for mapping product goals to metrics. The post-launch guide explains HEART and suggests metrics by feature type; you can use its table when choosing metrics in step 2.

• User interviews: what to ask to get honest answers

What is the difference between leading and lagging indicators?

In short, a leading indicator tells you whether people use the feature and can use it, and you can read it within a few weeks. A lagging indicator tells you whether that produced a business result, such as revenue or repeat purchases, and it takes longer.

When choosing a feature's primary metric, this article suggests preferring a leading indicator that sits close to the behaviour, because the team can influence it and read it within a reasonable window. Still record the lagging indicator on the sheet so you can follow it, but avoid making it the only number that decides whether a small feature succeeded. The full explanation, and how to read both after launch, is in the feature measurement guide.

• Leading and lagging signals when measuring a feature

What is a funnel, and where do users drop off?

Some PO job posts mention reading funnels and cohorts. They sound like a data analyst's job, but the basic ideas are simple.

A funnel is a sequence of steps people go through to finish a task, for example: start checkout, enter address, choose payment, place order. Amplitude's documentation describes a funnel as a defined path through the product, where a user counts as converted when they trigger the chosen events in the order the funnel defines. Some people drop off at each step. A funnel shows you where the biggest drop happens, so you know where to look.

Reading the table: of 4,000 people who started checkout, 2,106 placed an order, about 52.7%. The address step loses 1,400 people, far more than the next two steps combined. That is why the pre-fill feature targets step 2, not step 4.

Two things to watch when reading a funnel. First, always ask what the rate is divided by: the 65% here is relative to the previous step, while 52.7% is relative to the start. Second, a funnel tells you where people drop off, not why. For the why, look at support notes, watch people use the product, or talk to them.

A cohort is a group of users who share a property or performed a similar sequence of actions; Mixpanel defines cohorts in exactly those two ways. One use is grouping by time: for example, take everyone whose first purchase fell in the same week, then see what share of each group buys again within 30 days. Comparing this week's group with last week's lets you compare like with like, instead of mixing new and returning customers into one number. For Bếp Xinh's feature, “returning customers on mobile” is itself a cohort: the primary metric is measured on exactly that group.

If you have read access to the data, counting people at each funnel step is one of the first queries worth learning.

• Does a PO need to know SQL?

Guardrail metric examples by feature type

A guardrail answers one question: if the team pushes the primary number up at any cost, what could break? The table below is this article's suggestion for discussion with your team, not a required list.

A guardrail needs a limit, not a growth target. For example: “orders returned for a wrong address stay at or below 2%”. If it crosses the limit, the team stops to investigate even when the primary metric looks great.

Example: a metric sheet for address pre-fill

Back to Bếp Xinh. This is the metric sheet the team could agree during refinement, based on the funnel above. The shop and every number are assumptions for illustration.

Time window: four weeks after the feature is on for all returning customers, reviewed on a date already in the calendar. If those four weeks overlap a big sale such as 11.11 or 12.12, note it on the sheet, because traffic and behaviour during a sale may differ from normal days.

A quick sanity check on the target. If the address pass rate reaches 75% and the next two steps stay at 90%, then with the same 4,000 checkout starts: 3,000 pass the address step, 2,700 choose a payment method and 2,430 place an order. That is 324 more orders in four weeks than today's 2,106. The calculation assumes the later steps do not change. It helps sales see what the target means, but total orders is still a lagging signal and does not replace the primary metric.

Decision rules, written before any results. The team agrees with the data owner that a change of less than 2 percentage points, up or down, counts as no change. The 2 points is also an assumption for this example; for your own user numbers, ask the data owner what size of change is large enough not to be mistaken for noise.

• Primary metric at 75% or above and both guardrails within limits: keep the feature and consider a next step, such as address suggestions for new customers.
• Primary metric from 67% to below 75% (up at least 2 points but short of target): split the funnel by operating system or by number of saved addresses to find the group still dropping off, then decide whether to keep iterating or stop.
• A guardrail crosses its limit, whatever the primary metric shows: switch off pre-fill for the affected group and find the cause before switching it back on, even if the primary metric is rising.
• Primary metric above 63% and below 67% (a change under 2 points, counted as no change): go back and find out why people drop off at the address step; the problem may not be retyping the address at all.
• Primary metric at 63% or below (down 2 points or more): first check that the “address completed” event is still recorded correctly. If the data is right, switch pre-fill off and find out why more people drop off before switching it back on.

The “checkout started” and “address completed” events already exist, which is how the team got the 65% baseline. But the feature changes the very step being measured: with a pre-filled address, customers can move on without typing anything, and the “address completed” event may stop firing. So the story's acceptance criteria should include a line checking that both events are still recorded correctly, including when customers use a pre-filled address, so QA checks the data as well as the interface. If the event stops firing, the primary metric will drop even though customers get through the step faster.

• What are acceptance criteria? Writing them so QA can test

Is this metric good enough?

Before writing a metric into the document, go through the five questions below. If the answer is yes, move on. If the answer is no or not yet, do the action listed with that question and ask again.

When you report results later, even if an AI tool helped compile the numbers, check them against exactly the formula and user group written on the sheet.

• Numbers from AI analysis: what to check before reporting

A one-page metric definition sheet

Copy the template below into your team's document, fill it in during refinement, and have the data owner read it too. The goal part of the sheet can go straight into the “Goals and how we measure” section of a one-page requirements document.

Feature: [name]
Owner: [PO name] | Data owner: [name]
Date agreed: [date]

1. PROBLEM
[Who] struggles with [what], seen in [where, what evidence].

2. EXPECTED BEHAVIOUR
With this feature, [who] will [do what differently] in order to [achieve what].

3. PRIMARY METRIC
Name: [name]
Formula: [numerator] divided by [denominator]
User group: [group]
Baseline: [number], from [period]
Target: [specific number]

4. GUARDRAIL METRICS (1 to 2)
Name: [name] | Formula: [formula] | Baseline: [number] | Limit: [number]
Name: [name] | Formula: [formula] | Baseline: [number] | Limit: [number]

5. TIME WINDOW
Measure from [date] to [date]. Review date: [date].
Events that could distort the numbers: [sale, campaign, other changes]

6. DATA SOURCE
Event or table: [name]
Tool: [name]
Extra tracking needed: [yes / no, what]

7. LAGGING SIGNAL TO FOLLOW
[metric, e.g. total orders or repeat purchase rate]

8. DECISION RULES
Hits target, guardrails within limits: [action]
Change that counts as no change: under [number] points, agreed with the data owner
Up by at least that much but short of target: [action]
Change smaller than that, counted as no change: [action]
Down by at least that much: [check the data, then action]
A guardrail crosses its limit: [action]

• What is a PRD? Writing a one-page requirements document

Common mistakes when setting success metrics

• Writing “increase conversion” with no number. Without a baseline and target, any increase can be called a success.
• Choosing a metric too far from the feature. Monthly revenue moves with promotions, seasons and ads; a small feature rarely makes a readable difference to it.
• Setting five or six primary metrics. With many numbers, it is easy to pick the one that looks good after launch. Keep one primary metric; the rest are guardrails or tracking.
• Forgetting guardrails. The address pass rate can rise because an old, wrong address was pre-filled. Without measuring wrong deliveries, the team never sees that cost.
• Not asking the data owner first. Some perfectly sensible metrics are not tracked anywhere yet. Asking early puts the tracking work into scope.
• Changing the target after the results. If you really must change it, record the reason and date on the sheet, and report against both the old and the new target.

Summary

• Success metrics are the numbers a team agrees in advance to judge whether a feature produced the expected behaviour change.
• The chain to agree: problem, expected behaviour, one primary metric, one or two guardrails, baseline, target, time window, data source.
• A funnel shows which step people drop off at. A cohort lets you compare the same kind of users with each other.
• Guardrails have limits, not growth targets. If one crosses its limit, stop and investigate even when the primary metric looks good.
• Write the decision for each outcome before launch, so the goalposts cannot move.

I am writing more PO situations in this same format, with templates you can use straight away. If you would like the next one by email, subscribe to the newsletter below.

• Get new articles by email

Frequently asked questions

What are success metrics?

Success metrics are the numbers a team agrees in advance to judge whether a feature produced the expected change in user behaviour. Each one needs a formula, a baseline, a target, a time window and a data source.

What is a guardrail metric?

A guardrail metric is a number the team does not want to get worse while chasing the primary metric, such as page load time, error rate or the share of wrongly delivered orders. It does not need to improve; it only needs to stay within an agreed limit.

How many success metrics should a feature have?

One primary metric that answers whether it worked, plus one or two guardrail metrics. With more than that, it becomes easy to pick whichever number looks best after launch.

Sources to check

• Google Research: Measuring the User Experience on a Large Scale (HEART)
• Amplitude: Build a funnel analysis
• Mixpanel: Cohorts
• Microsoft Research: Patterns of trustworthy experimentation, experiment stage

Related articles

• Measuring a feature after launch: did it help users?: Shipping is output, not success. This guide shows how to set the expected behaviour change, a measure, a baseline, a target and a time window before building, then read the results to decide whether to iterate, expand, measure longer or remove.
• Does a Product Owner need to know SQL?: You do not need to become a data analyst, but you should know how to turn curiosity into a question you can test.
• Using Claude as a PO: where it helps, what you still do: AI drafts fast. This article shows four mistakes to check and a checklist to use before a document reaches the team.
• What is a user story? How to write one, with examples: The “As a…, I want…, so that…” template is the easy part. This article shows how to pick a real user, a real problem and a small enough slice, with before and after examples.