Measuring AI ROI honestly: a method that survives the finance review

Honest AI ROI is measured against a baseline captured before you build, counts the full cost including tokens, operations and change management, and attributes benefit through a holdout or phased rollout rather than a before-and-after comparison. Hard benefits (hours removed, revenue added, errors avoided) are counted in money; soft benefits are tracked but reported separately. Anything else is a story, and finance teams can tell the difference.

TL;DR

• Capture the baseline before the build, on the exact metric the AI is meant to move.
• Separate hard benefits (cash, hours, errors) from soft ones (satisfaction, speed to insight) and never blend them into a single headline number.
• Count all costs: build, tokens and inference, integration, monitoring, on-call, retraining, training people and changing process.
• Attribute with a holdout group or phased rollout; a before-and-after chart proves very little.
• Track leading indicators weekly and lagging ones quarterly, and report time-to-value alongside the ratio.

What should you measure before you build?

The most common ROI failure is a missing baseline. Once the system is live, nobody remembers precisely how long the old process took, so the number gets reconstructed from memory, always generously.

Before writing code, define the one or two metrics the project exists to move and measure them for a few weeks: handling time per ticket, days to close a claim, cost per qualified lead, error rate per thousand invoices. Record how they were measured so the post-launch measurement is identical. This is often the first deliverable in our AI consulting and strategy work, and sometimes the baseline shows the problem is smaller than assumed, which is cheaper to learn in week one.

Hard benefits or soft benefits: which count?

Both matter; only one goes in the numerator.

• Hard benefits convert to money with a defensible formula: hours removed multiplied by a loaded hourly rate, but only if those hours are actually redeployed; incremental revenue attributable to the system; avoided penalties, refunds, rework or churn; reduced vendor or licence spend.
• Soft benefits are real but resist conversion: employee satisfaction, faster answers, better decisions, capacity for work that never got done. Track and report them; do not invent a cash value for them.

A useful discipline: if finance would not accept the formula for a non-AI project, do not use it here.

What does the full cost include?

The build is rarely the biggest number over three years. Count:

• Build: engineering, data preparation, evaluation set creation, security review.
• Run: tokens or inference compute, storage, hosting, third-party APIs, measured monthly because usage grows.
• Operate: monitoring, on-call, prompt and model updates, retraining, incidents. Our MLOps and infrastructure team typically sees this at 15–30% of build cost per year for a maintained system.
• Change management: training staff, updating procedures, the productivity dip during rollout, and manager time making the new process stick.
• Risk reserve: a modest allowance for rework when the first version misses.

Leaving out the last three is how projects that "paid back in four months" get quietly retired in month ten.

How do you attribute the benefit?

Before-and-after comparisons are contaminated by everything else that changed: seasonality, a new hire, a price change. Three better options, in order of rigour:

1. Randomised holdout. Route a share of cases, say 10–20%, through the old process for the first quarter and compare outcomes directly. Cleanest evidence, and often the cheapest.
2. Phased rollout. Launch by team, region or product line in sequence, and use the not-yet-launched groups as controls.
3. Matched comparison. When neither is possible, compare with a similar team or period and say plainly the estimate is weaker.

Decide the analysis before launch and stick to it. Our data analytics and predictive AI team sets up the measurement in the same sprint as the build; retrofitting is far harder.

Which indicators do you watch, and when?

Leading indicators tell you within weeks whether the system is used and doing the job; lagging indicators tell you within quarters whether that became money. Report both, plus time-to-value (the date cumulative net benefit crossed zero) as prominently as the ratio. A 4x return in year three is a different investment from 1.5x in month four.

What does an honest calculation look like?

The following figures are invented for illustration and are not from a specific client.

A team of twelve handles 4,000 supplier invoices a month. Baseline, measured over six weeks: 9 minutes per invoice, 2.1% requiring rework, all-in loaded cost of the team's time at 30 per hour. An AI automation system is introduced to extract, validate and route invoices, with 15% of invoices held out for a quarter.

• Treated invoices: 3.5 minutes of human time each; rework 1.2%.
• Hard benefit: 5.5 minutes × 3,400 invoices × 12 months ≈ 3,740 hours a year, worth about 112,000, but only 70% of that time is genuinely redeployed, so the team counts about 78,000. Rework reduction adds an estimated 9,000 in avoided correction time.
• Full annual cost: 60,000 build amortised over three years (20,000), 14,000 inference and hosting, 12,000 operations, 8,000 first-year change management: about 54,000 in year one, roughly 46,000 in later years.
• Year-one net: around 33,000; time-to-value: month eight. The ratio improves in year two as adoption rises.

Modest, defensible, and the kind of number a CFO nods at.

By contrast, the red flags of an inflated ROI:

• Benefits stated in hours saved with no evidence the hours were redeployed.
• A baseline reconstructed after launch, or no baseline at all.
• Before-and-after only, during a period when other things changed.
• Soft benefits converted to money with a formula nobody would accept elsewhere.
• Costs limited to the build, with tokens, ops and change management omitted.
• A ratio quoted with no time horizon and no time-to-value.
• Adoption not reported; a system nobody uses returns nothing.

Key takeaways

• Measure the baseline before you build, on the metric the project is meant to move.
• Put hard benefits in the numerator; report soft benefits alongside, not blended in.
• Count build, run, operate, change management and a risk reserve, over three years, not one.
• Attribute with a holdout or phased rollout decided before launch.
• Report leading indicators weekly, lagging ones quarterly, and time-to-value with the ratio.

FAQ

How long should we wait before calculating ROI?

Report leading indicators (adoption, handling time, override rate) from the first month, and a first honest ROI estimate once you have a full quarter of holdout or phased data. Financial lagging indicators such as churn or revenue often need two quarters. A ratio published in the first few weeks was almost always computed from assumptions rather than measurements, and will need walking back.

What if we cannot run a holdout?

Use a phased rollout by team or region so the later groups act as controls, and if even that is impossible, use a matched comparison and say clearly that the estimate is weaker. Lean harder on internally measurable leading indicators such as handling time from system logs. A weaker but honest estimate is far more useful than a precise-looking one built on a contaminated before-and-after comparison.

Should soft benefits ever be included in the ROI figure?

Report them, but keep them out of the headline ratio unless you have a formula the finance team has already accepted for other investments. Faster decisions, happier staff and freed-up capacity are genuine and often why a project is worth doing; monetised on a guess, they cost the whole figure its credibility. If you would like help designing a measurement plan before your next AI build, get in touch.