Measuring a feature after launch: did it help users?
Shipping is output, not success. This guide shows how to set the expected behaviour change, a measure, a baseline, a target and a time window before building, then read the results to decide whether to iterate, expand, measure longer or remove.
Written by Phan Lam Duy Khoi
Before building, write down what users will do differently, which number shows it, today's value, the target and when you will check. After launch, read early and late signals together with qualitative feedback, then decide to iterate, expand, measure longer or remove.
The feature has launched. Now what?
Your team has just released a “Reorder” button in the app of an online coffee bean shop. The ticket is Done, the release is stable and there are no serious bugs. In the weekly meeting, your manager asks one short question: “Did this actually help customers?”
You open the dashboard and see plenty of numbers: app opens, orders, revenue, button taps. Each of them has moved up or down a little. You are not sure where to look, or which change, if any, came from the feature.
This is an illustrative situation; the shop and every number in this article are assumed. The article walks through how to answer your manager with evidence: what to prepare before building, how to read the numbers after launch, how to combine them with what users tell you, and how to use the result to decide the next step.
What is the difference between output and outcome?
Output is what the team produces: a screen, a button, a release. Outcome is the change that happens for users because of it: they do something faster, make fewer mistakes, or come back more often. A feature can match the design, ship on time and pass every test, and still change nothing for users.
This is hard for new POs for several reasons.
- Done feels finished. The Definition of Done confirms the work meets the quality bar for release, but it does not tell you whether users gained anything.
- Nobody agreed up front what to measure. If you start looking for numbers at launch, it is easy to pick the one that looks best rather than the one that matters most.
- There is no starting number. If you do not know the value before the change, you cannot say whether things improved after it.
- Many things change at once. Promotions, shopping seasons and ad campaigns can all push numbers up or down, with nothing to do with your feature.
- Bad results are uncomfortable. After the effort of building, everyone wants the feature to succeed, so it is easy to look only for supporting evidence.
The good news is that much of the measuring work happens before building, not after launch. Prepare well, and answering your manager means opening a single page of notes.
How do you measure a feature after launch?
One simple way has four steps. The first two happen before the team starts building; the last two happen after launch.
- Set the expected change: Who will do what differently
- Choose the measures: Baseline, target, time window
- Watch the signals: Numbers plus user feedback
- Review and decide: Iterate, expand, measure longer, remove
Every decision in step 4 needs a new checkpoint. Go back to step 2 for the next round.
Step 1: write the expected behaviour change as one sentence: “Once this exists, [who] will [do what differently] so that [they achieve what]”. The sentence is about users, not about the button. If the team cannot write it, the problem is probably not clear yet, and it is worth going back to learn more before building.
Step 2: choose two or three measures tied to that sentence. Each needs four parts: the name and how it is calculated, the baseline (the value before the feature exists), the target, and the date you will check. Ask whoever owns the data whether it can be measured and where it comes from. The main measure, the number that answers the step 1 sentence most directly, goes in the “Goal and measure” section of the one-page requirements document. Supporting measures go in the post-launch review template at the end of this article.
Step 3: after launch, check the numbers on the agreed schedule, not every hour. Alongside them, read support tickets and in-app feedback, and talk to a few users. The numbers tell you what is happening; users help you understand why.
Step 4: when the date arrives, sit down with the team, compare results with targets and decide the next step. Write the decision on one page so anyone can find it later.
Which metrics should you use for each type of feature?
Researchers at Google proposed the HEART framework, five categories of user experience metrics: Happiness (satisfaction), Engagement (how involved people are), Adoption (new people starting to use it), Retention (people coming back) and Task success (finishing the task quickly and with few errors). The original paper itself says it is not always appropriate to use every category; the point is to decide deliberately which ones fit the goal.
The paper also pairs the framework with a three-step process: Goals, Signals (behaviour or feelings that show the goal is being met) and Metrics (specific numbers that track those signals). Steps 1 and 2 above are a shortened version of it. Satisfaction cannot be read from taps, and Amplitude notes that Happiness is typically measured through surveys.
The table below is this article's suggestion for some common types of feature. Treat it as a starting point for a conversation with your team, not a fixed answer.
| Feature type | Expected behaviour | Early signals | Late signals |
|---|---|---|---|
| New feature for a repeated task, such as reordering | Customers do a familiar task faster | Share of users who try it, share who finish after tapping | Share of customers who buy again within a set period |
| Shortening an existing flow, such as checkout | Fewer people drop out halfway | Completion rate, time to complete, input errors | Completed orders per person who starts checkout |
| Internal tool for operations, such as a returns screen | Staff handle cases faster and with fewer errors | Time per case, number of manual corrections | Customer waiting time, complaints about returns |
| Notifications or reminders | Users come back to finish something | Open rate, rate of turning notifications off | Share of users returning within a week or month |
| Search and filters | Users find what they need faster | Searches with no results, clicks on results | Share of searches that lead to a purchase or use |
For internal tools, users are staff who use the tool as part of their job, so engagement means little. The HEART paper gives a similar example for enterprise products. There, finishing the task quickly and correctly is the more useful thing to measure.
What are leading and lagging signals?
Amplitude distinguishes two kinds of indicator. Leading indicators help predict future results and can guide day-to-day work. Lagging indicators reflect results that have already happened, such as revenue, and are used to assess success.
For a new PO, a practical reading is this. Early signals answer “are people using it, and can they use it?” and can be read after a few weeks. Late signals answer “did it produce the result we wanted?” and take longer. Watch only late signals and you wait a long time before spotting a problem. Watch only early signals and you may celebrate lots of taps while the real result has not moved.
Set the time window around how often people use the product. Coffee bean customers may reorder after a few weeks, so measuring repeat purchases after one week is too early. A daily-use app can show signals much sooner.
To get the numbers, you can ask a data colleague, use the team's analytics tool, or write a few simple queries yourself if you have read access to the data.
Example: the Reorder button after four weeks
Back to the coffee bean shop from the start. Every number below is assumed, to show how to read results, and does not come from any real shop.
Before building, the team wrote the expected change: “Returning customers will reorder their usual order more easily, and so buy again more often.” They chose three measures, matching the template at the end: two early signals, checked after four weeks, and one late signal, checked after eight weeks because customers need time to run out of coffee. The main measure in the requirements document is the share of customers buying again within 30 days.
| Measure | Baseline | Target | Result | How to read it |
|---|---|---|---|---|
| Returning customers who use Reorder (after 4 weeks) | No feature yet | 25% or more | 31% | Met. Customers find and try it. |
| Orders completed after tapping Reorder (after 4 weeks) | No feature yet | 80% or more | 62% | Not met. Many tap but do not finish. |
| Customers buying again within 30 days (after 8 weeks) | 18% | 21% | 20% | Inconclusive. A sale ran during the period. |
Looking only at usage, the team would report success. The low completion rate shows something is in the way. The numbers do not say what, so the PO reads the 14 related support tickets and calls five customers who tapped Reorder but did not finish.
All five calls and most tickets point to the same thing: when one item in the old order is out of stock, the whole reorder fails and the customer has to empty the basket and start again. This is an error case the team did not consider when writing the acceptance criteria.
Repeat purchases rose from 18% to 20%, but a store-wide sale ran in the middle of the period. A before-and-after comparison can be affected by other factors, so the team does not credit the rise to the feature. They plan to measure again in a month with no major promotion.
Decision: iterate. The team lets Reorder drop out-of-stock items, tell the customer clearly, and place the rest. The completion target stays at 80% or more, checked four weeks after the fix ships. The idea of expanding into scheduled subscriptions waits until reordering works smoothly.
When should you iterate, expand or remove a feature?
When the review date arrives, go through four questions in order. A yes moves you forward. A no means doing the action beside that question.
The planned window for early signals has ended
- Are people using the feature at the target level? Find out why: they cannot find it, do not need it, or find it hard. If they do not need it, consider removing it.
- Can people complete the task easily? Fix the specific point where they get stuck, then measure again against the same target.
- Was the late-signal window a normal period, free of major promotions or seasonal peaks? Keep the feature, note the factor and measure again in a normal period. Do not expand yet.
- Have late signals moved in the expected direction? Keep the feature and revisit the assumption in the expected-change sentence. Do not expand yet.
Expand the feature, or move the team's effort to the next problem.
Every option has a cost. Iterating takes more team time and pushes back other backlog items. Expanding increases what must be maintained and tested. Waiting to measure more means your manager does not get a final answer yet. Removing a feature means warning current users, cleaning up code, and possibly upsetting a small group who liked it. Keeping a little-used feature also costs something: it still has to be tested and supported in every release.
When you present to your manager, give both the recommended decision and its cost, with the evidence: which number, against which target, and what users said.
How do you combine numbers with user feedback?
Numbers tell you how many people did what. Qualitative feedback tells you why. With only one of the two, it is easy to misread what is happening.
- Support tickets and in-app feedback: filter by the feature name or related keywords, and read the originals instead of only counting them.
- A few short conversations: pick people who used the feature and people who gave up. Ask what they did the last time, not whether they like the feature.
- Support and operations staff: they hear complaints every day and may see a problem before it shows up on a dashboard.
- One short survey question right after use: useful when the goal is satisfaction, which taps cannot show.
Five calls are not enough to draw conclusions about every customer. They are enough to explain an unusual number and suggest what to fix. After the fix, the numbers will tell you whether that explanation was right.
A one-page post-launch review template
Use this for each feature. Fill in the top half before the team starts building and the bottom half when the review date arrives. Copy it into your team's document or wiki.
Feature: [name] Launch date: [date] Review owner: [name] PART 1: FILL IN BEFORE BUILDING Expected change: Once this exists, [who] will [do what differently] so that [they achieve what]. Measure 1 (early signal): [name, how it is calculated] | Baseline: [value] | Target: [value] | Review date: [date] Measure 2 (early signal): [name, how it is calculated] | Baseline: [value] | Target: [value] | Review date: [date] Measure 3 (late signal): [name, how it is calculated] | Baseline: [value] | Target: [value] | Review date: [date] Data source: [tool, table, person who owns the number] Events that could skew the numbers during the window: [promotions, seasons, campaigns, other changes] PART 2: FILL IN AT THE REVIEW DATE Result for each measure against its target: [met / not met / inconclusive, with the number] Qualitative feedback: [tickets, calls, survey: what came up most often] Other factors during the window: [yes / no, what] What the team learned: [1 to 3 points] Decision: [iterate / expand / measure longer / remove] Reason and cost of the decision: [short] Next checkpoint: [measure, target, date]
The line about events that could skew the numbers is easy to leave blank, yet it is what stops you taking credit that is not yours. Ask marketing and sales for the promotion calendar before launch.
Common mistakes when measuring a feature
- Treating launch as success. Reporting “released on time” with nothing about users.
- Choosing measures after seeing the results. It is then very easy to pick the number that looks good. Set measures and targets before building.
- Measuring too much. Ten numbers on a dashboard do not help a decision. Two or three tied to the expected behaviour are enough.
- Counting only taps. Many taps do not mean people finished the task. Always pair usage with a completion measure.
- Crediting every change to the feature. A rise during a sale month may not come from the feature. Record other events and compare with a similar period.
- Comparing users with non-users and concluding. People who try a new feature early may already be more loyal customers. A gap between the groups does not prove the feature made the difference.
- Reviewing without deciding. Every review should end with a decision and a new checkpoint.
Summary
- Launch is output. Outcome is users doing something differently and getting what they need because of the feature.
- Before building, write the expected behaviour change in one sentence, then pick two or three measures, each with a baseline, a target and a date.
- Read early signals (do people use it, can they finish) alongside late signals (do they come back, does it produce the result), timed to how often the product is used.
- Numbers show what happened. Support tickets and a few conversations help explain why.
- Before-and-after comparisons can be affected by other factors such as promotions or seasons. Record them before drawing conclusions.
- Every review ends with a decision, its reason and its cost: iterate, expand, measure longer or remove.
I am writing more PO situations in this same format, with templates you can use straight away. If you would like the next one by email, subscribe to the newsletter below.
Frequently asked questions
What is the difference between output and outcome?
Output is what the team delivers, such as a launched feature. Outcome is the change in users' behaviour or results because of it. Shipping on time is output; users finishing a task faster or coming back more often is outcome.
Which metrics should I use for a new feature?
Start from the behaviour you expect to change: how many people use the feature, whether they complete the task, and whether they come back. Two or three measures tied to the goal are enough; you do not need to track everything.
How long after launch should I evaluate a feature?
Set the time window before building. Early signals such as usage can be read after a few weeks, while late signals such as repeat purchases or revenue need longer, depending on how often people use the product.
Sources to check
Related articles
- Success metrics: how to set them before you build: How to agree a feature's success metrics before the team starts building: one primary metric, guardrail metrics, a baseline, a target, a time window and a data source, with a one-page definition sheet.
- Does a Product Owner need to know SQL?: You do not need to become a data analyst, but you should know how to turn curiosity into a question you can test.
- Using Claude as a PO: where it helps, what you still do: AI drafts fast. This article shows four mistakes to check and a checklist to use before a document reaches the team.
- What is a user story? How to write one, with examples: The “As a…, I want…, so that…” template is the easy part. This article shows how to pick a real user, a real problem and a small enough slice, with before and after examples.