Numbers from AI analysis: what to check before reporting
How a PO checks numbers an AI tool calculated: reconcile totals, check the denominator and the period, recompute a few key figures, and label what is still uncertain.
Written by Phan Lam Duy Khoi
Before an AI-calculated number goes into a report, reconcile totals with the source system, check the numerator, denominator and period, compare row counts before and after filters, and recompute the two or three most important figures yourself. If a number cannot be traced back to the source data, leave it out.
The numbers look great
This is an illustrative situation. You are the PO at Gốm Mộc (a fictional name), an online shop selling ceramics. Tomorrow morning you have to send a report on the “Ceramics Week” campaign that ran from 12 to 18 October 2026. You have two files: the order export for that week and a file of website sessions. Your company allows AI tools for data with customer details removed, so you give both files to an AI tool and ask for a summary of the results.
A few seconds later you have a neat summary: “The campaign had 362 orders, revenue of VND 181 million and a conversion rate of 36.2%, more than 10 times September thanks to the promotion.” The wording is clear, the numbers have decimals, and it sounds professional. You nearly paste it straight into the report for the head of sales.
None of those four figures can be used as they stand. Internal test orders were counted, the conversion rate used the wrong denominator, and “thanks to the promotion” is a conclusion the data does not prove. This article covers how to check numbers an AI tool analysed before you put your name under the report.
Why are AI-analysed numbers hard to check?
Errors in text are easier to spot because you understand what you read. Errors in numbers are different: 36.2% and 4.4% both look plausible if you do not know how they were calculated. A number with a decimal place even feels carefully worked out.
The US National Institute of Standards and Technology (NIST) calls the confident production of false content “confabulation”, also known as “hallucination”. NIST also notes something worth remembering for anyone working with numbers: AI models sometimes lay out the logical steps behind an answer even when the answer itself is wrong. A coherent explanation does not prove the number is right.
Nielsen Norman Group (NN/g) explains the root of the problem: AI tools produce what is statistically likely, and cannot verify that what they write is true. A confident tone is not evidence.
There is also a reason on the human side. NIST lists people relying too much on machine output (over-reliance, automation bias) as a risk of its own. When you are in a hurry and the number looks good, it is easy to want to believe it. The report carries your name, so the responsibility for each number is yours, not the tool's.
Where do AI tools get numbers wrong?
The table groups the kinds of error worth looking for when you read an analysis done by AI. All examples are illustrative. The “Quick check” column is something you can do in a few minutes with a spreadsheet.
| Error type | Example | Quick check |
|---|---|---|
| Arithmetic slip | Product category shares add up to 107% | Add up the parts and compare with the total |
| Wrong denominator | Orders divided by sessions with an add-to-cart, then called the conversion rate | Ask: divided by what? Does it match the team's definition? |
| Double counting | An order with three items sits on three rows and is counted as three orders | Count distinct order IDs and compare with the row count |
| Mixed periods | Comparing a promotion week with a whole month, or mixing order date with delivery date | Write down the exact dates and which date field is used |
| Silently dropped rows | Orders missing a sales channel are excluded and nobody mentions it | Compare row counts before and after each filter |
| Misread column | Summing the list price column instead of the amount actually paid | Open the source file and check which column was used |
| Invented figure | “Each customer bought 1.8 items on average” when the file has no quantity column | Point to the source row and column for the number |
| Correlation taken as cause | “Revenue rose thanks to the promotion” when that week also had a public holiday | Rewrite it as an observation and list other things happening at the same time |
An arithmetic slip is an error in the calculation itself. A wrong denominator, double counting and mixed periods are definition errors: the arithmetic may be right, but it counts the wrong thing. Silently dropped rows and a misread column are input errors: a correct calculation on the wrong set of data. The last two are errors of fact and interpretation. An invented figure is the clearest case of what NIST calls confabulation, and it is hard to catch unless you ask “where does this number come from?”
Many of these errors are not unique to AI. People writing SQL or building spreadsheets make exactly the same ones, such as a JOIN that duplicates rows. The difference is that when you do the work yourself, you see every step. When AI does it, you only see the final result.
How do you check numbers an AI tool analysed?
One practical approach is to go through four steps in order, before any number leaves your computer. All you need is a spreadsheet and access to the source system.
- Reconcile totals: Row counts and totals match the source system
- Check definitions: Numerator, denominator and period as the team uses them
- Recompute: The two or three most important figures, in a spreadsheet
- Label in the report: Source, period, method and what is uncertain
If any step does not match, go back to the source file and find the gap. Do not pick the nicer number.
Step 1, reconcile totals. Open the source system, for example the order admin page, and get the order count and total revenue for the same period. Compare them with the AI's figures. At the same time, note the file's row count before filtering and after each filter. If rows disappear, you should be able to say why: cancelled orders removed, test orders removed, or rows dropped that nobody meant to drop.
Step 2, check definitions. For every rate, write two lines: what is the numerator, what is the denominator. Then compare with the definition your team uses. “Conversion rate” in one team may be completed orders divided by sessions; in another, buyers divided by visitors. Check the period too: which dates, whether it uses order date or delivery date, and whether the period you compare against is the same length. If your team has no shared definitions yet, the article on measuring a feature after launch discusses how to choose metrics.
Step 3, recompute. You do not need to recompute everything. Pick the two or three numbers that will drive a decision or be quoted most, and calculate them yourself in a spreadsheet. You can ask the AI tool to show its calculation, but treat that as a list to check, not as proof: take each step, apply it to the real data and see whether you get the same result. To finish this step, set the number next to what you know about the product. A small shop with a 36% conversion rate needs a very convincing explanation.
Step 4, label in the report. Every important number should come with its source, period and method in one short line. Where something is uncertain, say so plainly: “sessions come from the analytics tool and do not yet exclude internal visits”. Readers need to know how much to trust a number to make a decision, and labelling it now saves you a correction later.
Is this number reliable enough for the report?
When you are short on time, run each number through the questions below. A “yes” moves you forward; a “not yet” means doing the action beside it first.
A number calculated by an AI tool
- Can you point to the file and period this number comes from? Find the source and period. If you cannot, leave the number out.
- Do the main totals and row counts match the source system? Find the gap: test orders, cancelled orders, duplicate rows, dropped rows.
- Do the numerator, denominator and comparison period match the team's definition? Recalculate using the agreed definition, or ask the data owner.
- Have you recomputed it yourself and got the same result? Recompute it in a spreadsheet. If they differ, find out why before using either.
- Does the conclusion attached to it say only what the data shows? Rewrite it as an observation and turn the “why” into a hypothesis to test.
Put it in the report, with source, period, method and a note on what is uncertain.
The last question is easy to skip because it is not about the number. But “up thanks to the promotion” is what readers remember and use to decide budgets. Data showing two things happened at the same time does not mean one caused the other.
Example: is the conversion rate 36.2% or 4.4%?
Back to Gốm Mộc. Every number here is illustrative, for a fictional shop. After the four steps, you find these facts:
- The order file has 362 rows, one per order. Of these, 12 are internal test orders, flagged in the notes column. There were no cancelled orders that week. Completed orders: 362 minus 12 equals 350.
- The total of all 362 rows is VND 181 million. The 12 test orders at VND 500,000 each add up to VND 6 million. Real revenue: VND 175 million, matching the order admin page.
- The sessions file has 8,000 sessions for the week, of which 1,000 included an add-to-cart. To calculate the cart-to-order rate, this example assumes each order came from its own session with an add-to-cart.
- The team defines conversion rate as completed orders divided by sessions. In September, on the same definition, it was 3.5%.
| Line in the AI summary | What the check shows | Corrected version |
|---|---|---|
| The campaign had 362 orders | The row count includes 12 internal test orders | 350 completed orders |
| Revenue of VND 181 million | Does not match the admin page; the gap is exactly the VND 6 million of test orders | VND 175 million, matching the admin page |
| Conversion rate of 36.2% | It divided 362 orders by 1,000 add-to-cart sessions, so both numerator and denominator are wrong | 4.4% (350 orders divided by 8,000 sessions). Cart-to-order rate on its own is 35% (350 divided by 1,000, assuming one session per order) |
| More than 10 times September thanks to the promotion | Compares a wrongly defined number with a correctly defined one, a promotion week with a whole month, and claims a cause | This week 4.4%, September 3.5%. A promotion week and a whole normal month differ in length and conditions, so they are not directly comparable. To compare, use a normal week of the same length with the same definition. The data is also not enough to call the promotion the cause |
The biggest error is the denominator. The 36.2% is really 362 divided by 1,000, which is close to a different metric: the cart-to-order rate. That metric is useful, but it is not the conversion rate as the team defines it. If the report said 36.2%, readers would compare it with September's 3.5% and think the rate grew more than tenfold, when the correct figure is 4.4%, and even that is not directly comparable with the whole of September.
The corrected summary: “Ceramics Week (12 to 18 October 2026): 350 completed orders, revenue of VND 175 million, matching the admin page. Conversion rate 4.4% (350 orders from 8,000 sessions). Cart-to-order rate 35%, assuming each order came from one session. September's conversion rate was 3.5%, but this week had a promotion and a different length, so the two are not directly comparable; a normal week of the same length is needed for comparison. Sessions do not yet exclude internal visits.” It is less impressive, but every number traces back to the source data.
A number-check checklist for reports
Use this checklist for every report with numbers calculated by an AI tool, or by anyone, that you have not checked yourself. Copy it, paste it at the end of your draft report, and tick each item before sending.
Report: ... Data source (file, system, export date): ... Period: from ... to ..., based on ... date Reconcile [ ] Rows before filtering: ... / after filtering: ... / I know why rows were removed. [ ] Main totals (orders, revenue...) match the source system. Gap, if any: ... [ ] Test data, internal data and duplicate rows are excluded. Definitions [ ] Every rate states its numerator and denominator. [ ] Definitions match how the team uses them. [ ] The comparison period has the same length and the same definitions. Recompute [ ] I recomputed the two or three most important numbers in a spreadsheet and got the same result. [ ] I checked each step of the method against the real data, not just the explanation. [ ] The numbers are plausible given what I know about the product. If something is unusual, I found the reason. [ ] There is no number I cannot trace to a source row and column. Writing the report [ ] Conclusions separate observations from hypotheses about causes. [ ] Each important number states its source, period and method. [ ] What is still uncertain is noted. [ ] The data given to the AI tool is within what the company allows.
The item “no number I cannot trace to a source row and column” is the one that catches invented figures. If a number exists only in the summary and not in the data, it does not go into the report.
The last item concerns the step before any analysis: which data you may give an AI tool at all. The article on data you should not put into AI tools covers this in more detail.
Common mistakes when checking AI numbers
- Checking the arithmetic but not the input data. A correct division on data that includes test orders still gives a wrong number.
- Trusting the explanation instead of recomputing. A plausible chain of steps does not prove the answer is right.
- Asking the AI tool whether the number is correct and treating “yes” as confirmation. Confirmation has to come from the source system or the data owner.
- Picking the nicer number when two sources disagree. The gap tells you a lot about your company's data; find out why it exists.
- Removing the uncertainty notes because they make the report look weak. Readers need to know how far to trust a number to decide well.
- Checking the numbers carefully but leaving the claim about causes untouched. That claim is what readers take away and act on.
The same checks apply when AI summarises customer feedback into themes with counts. “22 customers complained about delivery” also needs to trace back to the original feedback.
Summary
- AI tools can state wrong numbers with full confidence, along with a plausible explanation. Tone is not evidence.
- Four steps: reconcile totals with the source system, check definitions (numerator, denominator, period), recompute the two or three key numbers, and label source and uncertainty in the report.
- For every rate, ask first: divided by what? A wrong denominator can turn 4.4% into 36.2%.
- If a number cannot be traced to a source row and column, it does not go into the report.
- Separate observation from cause. Two things happening together does not prove one caused the other.
I am writing more PO situations in this same format, with templates you can use straight away. If you would like the next one by email, subscribe to the newsletter below.
Frequently asked questions
Can I trust numbers an AI tool analysed?
Only after you check them. AI tools can state wrong numbers with full confidence, so reconcile totals with the source system and recompute a few key figures before they go into a report.
How do I quickly check a percentage an AI calculated?
Identify the numerator, the denominator and the period, then do the division yourself in a spreadsheet. If the denominator does not match the definition your team uses, the rate is not ready to use.
If the AI explained its calculation, do I still need to check it?
Yes. NIST notes that AI models sometimes give logical-looking steps even when the answer is wrong, so check each step against the real data.
Sources to check
Related articles
- Using Claude as a PO: where it helps, what you still do: AI drafts fast. This article shows four mistakes to check and a checklist to use before a document reaches the team.
- Can AI replace a Product Owner? What still needs a PO: A balanced answer for new POs: AI helps with drafting, summarising and suggesting, while accountability for product value and the decisions behind it stays with a person.
- What data should you keep out of AI tools at work?: Six kinds of data a PO should keep out of AI tools, how to filter each piece before pasting, an anonymised complaint example and a ready-to-use checklist.
- Using AI to summarise customer feedback: what to check: AI tools group feedback and name themes fast, but they can invent quotes, lose frequency and drop small groups of customers. This article shows each failure mode and how a PO checks a summary before using it for a decision.