[{"data":1,"prerenderedAt":439},["ShallowReactive",2],{"blog-measuring-ai-roi-honestly":3,"blog-related-\u002Fblog\u002Fmeasuring-ai-roi-honestly":409},{"id":4,"title":5,"author":6,"authorRole":7,"body":8,"cover":392,"coverAlt":393,"date":394,"description":395,"draft":396,"extension":397,"meta":398,"navigation":399,"path":400,"readingTime":401,"seo":402,"stem":403,"tags":404,"updated":407,"__hash__":408},"blog\u002Fblog\u002Fmeasuring-ai-roi-honestly.md","Measuring AI ROI honestly: a method that survives the finance review","William Ali","Founder & AI Specialist",{"type":9,"value":10,"toc":375},"minimark",[11,15,43,48,51,60,64,67,81,84,88,91,128,131,135,138,159,167,171,258,265,269,272,280,294,297,300,323,327,344,348,353,356,360,363,367],[12,13,14],"p",{},"Honest AI ROI is measured against a baseline captured before you build, counts the full cost including tokens, operations and change management, and attributes benefit through a holdout or phased rollout rather than a before-and-after comparison. Hard benefits (hours removed, revenue added, errors avoided) are counted in money; soft benefits are tracked but reported separately. Anything else is a story, and finance teams can tell the difference.",[16,17,18,24],"blockquote",{},[12,19,20],{},[21,22,23],"strong",{},"TL;DR",[25,26,27,31,34,37,40],"ul",{},[28,29,30],"li",{},"Capture the baseline before the build, on the exact metric the AI is meant to move.",[28,32,33],{},"Separate hard benefits (cash, hours, errors) from soft ones (satisfaction, speed to insight) and never blend them into a single headline number.",[28,35,36],{},"Count all costs: build, tokens and inference, integration, monitoring, on-call, retraining, training people and changing process.",[28,38,39],{},"Attribute with a holdout group or phased rollout; a before-and-after chart proves very little.",[28,41,42],{},"Track leading indicators weekly and lagging ones quarterly, and report time-to-value alongside the ratio.",[44,45,47],"h2",{"id":46},"what-should-you-measure-before-you-build","What should you measure before you build?",[12,49,50],{},"The most common ROI failure is a missing baseline. Once the system is live, nobody remembers precisely how long the old process took, so the number gets reconstructed from memory, always generously.",[12,52,53,54,59],{},"Before writing code, define the one or two metrics the project exists to move and measure them for a few weeks: handling time per ticket, days to close a claim, cost per qualified lead, error rate per thousand invoices. Record how they were measured so the post-launch measurement is identical. This is often the first deliverable in our ",[55,56,58],"a",{"href":57},"\u002Fservices\u002Fai-consulting-strategy","AI consulting and strategy"," work, and sometimes the baseline shows the problem is smaller than assumed, which is cheaper to learn in week one.",[44,61,63],{"id":62},"hard-benefits-or-soft-benefits-which-count","Hard benefits or soft benefits: which count?",[12,65,66],{},"Both matter; only one goes in the numerator.",[25,68,69,75],{},[28,70,71,74],{},[21,72,73],{},"Hard benefits"," convert to money with a defensible formula: hours removed multiplied by a loaded hourly rate, but only if those hours are actually redeployed; incremental revenue attributable to the system; avoided penalties, refunds, rework or churn; reduced vendor or licence spend.",[28,76,77,80],{},[21,78,79],{},"Soft benefits"," are real but resist conversion: employee satisfaction, faster answers, better decisions, capacity for work that never got done. Track and report them; do not invent a cash value for them.",[12,82,83],{},"A useful discipline: if finance would not accept the formula for a non-AI project, do not use it here.",[44,85,87],{"id":86},"what-does-the-full-cost-include","What does the full cost include?",[12,89,90],{},"The build is rarely the biggest number over three years. Count:",[25,92,93,99,105,116,122],{},[28,94,95,98],{},[21,96,97],{},"Build:"," engineering, data preparation, evaluation set creation, security review.",[28,100,101,104],{},[21,102,103],{},"Run:"," tokens or inference compute, storage, hosting, third-party APIs, measured monthly because usage grows.",[28,106,107,110,111,115],{},[21,108,109],{},"Operate:"," monitoring, on-call, prompt and model updates, retraining, incidents. Our ",[55,112,114],{"href":113},"\u002Fservices\u002Fmlops-ai-infrastructure","MLOps and infrastructure"," team typically sees this at 15–30% of build cost per year for a maintained system.",[28,117,118,121],{},[21,119,120],{},"Change management:"," training staff, updating procedures, the productivity dip during rollout, and manager time making the new process stick.",[28,123,124,127],{},[21,125,126],{},"Risk reserve:"," a modest allowance for rework when the first version misses.",[12,129,130],{},"Leaving out the last three is how projects that \"paid back in four months\" get quietly retired in month ten.",[44,132,134],{"id":133},"how-do-you-attribute-the-benefit","How do you attribute the benefit?",[12,136,137],{},"Before-and-after comparisons are contaminated by everything else that changed: seasonality, a new hire, a price change. Three better options, in order of rigour:",[139,140,141,147,153],"ol",{},[28,142,143,146],{},[21,144,145],{},"Randomised holdout."," Route a share of cases, say 10–20%, through the old process for the first quarter and compare outcomes directly. Cleanest evidence, and often the cheapest.",[28,148,149,152],{},[21,150,151],{},"Phased rollout."," Launch by team, region or product line in sequence, and use the not-yet-launched groups as controls.",[28,154,155,158],{},[21,156,157],{},"Matched comparison."," When neither is possible, compare with a similar team or period and say plainly the estimate is weaker.",[12,160,161,162,166],{},"Decide the analysis before launch and stick to it. Our ",[55,163,165],{"href":164},"\u002Fservices\u002Fdata-analytics-predictive-ai","data analytics and predictive AI"," team sets up the measurement in the same sprint as the build; retrofitting is far harder.",[44,168,170],{"id":169},"which-indicators-do-you-watch-and-when","Which indicators do you watch, and when?",[172,173,174,190],"table",{},[175,176,177],"thead",{},[178,179,180,184,187],"tr",{},[181,182,183],"th",{},"Benefit type",[181,185,186],{},"How to measure it",[181,188,189],{},"Leading or lagging",[191,192,193,205,216,227,237,248],"tbody",{},[178,194,195,199,202],{},[196,197,198],"td",{},"Time saved",[196,200,201],{},"Handling time per case from system logs, holdout vs treated",[196,203,204],{},"Leading (weekly)",[178,206,207,210,213],{},[196,208,209],{},"Cost avoided",[196,211,212],{},"Errors, refunds, penalties, rework per thousand cases",[196,214,215],{},"Lagging (monthly)",[178,217,218,221,224],{},[196,219,220],{},"Revenue added",[196,222,223],{},"Conversion or upsell rate in treated vs control group",[196,225,226],{},"Lagging (quarterly)",[178,228,229,232,235],{},[196,230,231],{},"Capacity released",[196,233,234],{},"Cases per person per week; backlog age",[196,236,204],{},[178,238,239,242,245],{},[196,240,241],{},"Quality and satisfaction",[196,243,244],{},"CSAT, QA scores, escalation rate on treated cases",[196,246,247],{},"Mixed",[178,249,250,253,256],{},[196,251,252],{},"Adoption",[196,254,255],{},"Share of eligible cases handled by the system; override rate",[196,257,204],{},[12,259,260,261,264],{},"Leading indicators tell you within weeks whether the system is used and doing the job; lagging indicators tell you within quarters whether that became money. Report both, plus ",[21,262,263],{},"time-to-value"," (the date cumulative net benefit crossed zero) as prominently as the ratio. A 4x return in year three is a different investment from 1.5x in month four.",[44,266,268],{"id":267},"what-does-an-honest-calculation-look-like","What does an honest calculation look like?",[12,270,271],{},"The following figures are invented for illustration and are not from a specific client.",[12,273,274,275,279],{},"A team of twelve handles 4,000 supplier invoices a month. Baseline, measured over six weeks: 9 minutes per invoice, 2.1% requiring rework, all-in loaded cost of the team's time at 30 per hour. An ",[55,276,278],{"href":277},"\u002Fservices\u002Fai-automation","AI automation"," system is introduced to extract, validate and route invoices, with 15% of invoices held out for a quarter.",[25,281,282,285,288,291],{},[28,283,284],{},"Treated invoices: 3.5 minutes of human time each; rework 1.2%.",[28,286,287],{},"Hard benefit: 5.5 minutes × 3,400 invoices × 12 months ≈ 3,740 hours a year, worth about 112,000, but only 70% of that time is genuinely redeployed, so the team counts about 78,000. Rework reduction adds an estimated 9,000 in avoided correction time.",[28,289,290],{},"Full annual cost: 60,000 build amortised over three years (20,000), 14,000 inference and hosting, 12,000 operations, 8,000 first-year change management: about 54,000 in year one, roughly 46,000 in later years.",[28,292,293],{},"Year-one net: around 33,000; time-to-value: month eight. The ratio improves in year two as adoption rises.",[12,295,296],{},"Modest, defensible, and the kind of number a CFO nods at.",[12,298,299],{},"By contrast, the red flags of an inflated ROI:",[25,301,302,305,308,311,314,317,320],{},[28,303,304],{},"Benefits stated in hours saved with no evidence the hours were redeployed.",[28,306,307],{},"A baseline reconstructed after launch, or no baseline at all.",[28,309,310],{},"Before-and-after only, during a period when other things changed.",[28,312,313],{},"Soft benefits converted to money with a formula nobody would accept elsewhere.",[28,315,316],{},"Costs limited to the build, with tokens, ops and change management omitted.",[28,318,319],{},"A ratio quoted with no time horizon and no time-to-value.",[28,321,322],{},"Adoption not reported; a system nobody uses returns nothing.",[44,324,326],{"id":325},"key-takeaways","Key takeaways",[25,328,329,332,335,338,341],{},[28,330,331],{},"Measure the baseline before you build, on the metric the project is meant to move.",[28,333,334],{},"Put hard benefits in the numerator; report soft benefits alongside, not blended in.",[28,336,337],{},"Count build, run, operate, change management and a risk reserve, over three years, not one.",[28,339,340],{},"Attribute with a holdout or phased rollout decided before launch.",[28,342,343],{},"Report leading indicators weekly, lagging ones quarterly, and time-to-value with the ratio.",[44,345,347],{"id":346},"faq","FAQ",[349,350,352],"h3",{"id":351},"how-long-should-we-wait-before-calculating-roi","How long should we wait before calculating ROI?",[12,354,355],{},"Report leading indicators (adoption, handling time, override rate) from the first month, and a first honest ROI estimate once you have a full quarter of holdout or phased data. Financial lagging indicators such as churn or revenue often need two quarters. A ratio published in the first few weeks was almost always computed from assumptions rather than measurements, and will need walking back.",[349,357,359],{"id":358},"what-if-we-cannot-run-a-holdout","What if we cannot run a holdout?",[12,361,362],{},"Use a phased rollout by team or region so the later groups act as controls, and if even that is impossible, use a matched comparison and say clearly that the estimate is weaker. Lean harder on internally measurable leading indicators such as handling time from system logs. A weaker but honest estimate is far more useful than a precise-looking one built on a contaminated before-and-after comparison.",[349,364,366],{"id":365},"should-soft-benefits-ever-be-included-in-the-roi-figure","Should soft benefits ever be included in the ROI figure?",[12,368,369,370,374],{},"Report them, but keep them out of the headline ratio unless you have a formula the finance team has already accepted for other investments. Faster decisions, happier staff and freed-up capacity are genuine and often why a project is worth doing; monetised on a guess, they cost the whole figure its credibility. If you would like help designing a measurement plan before your next AI build, ",[55,371,373],{"href":372},"\u002Fcontact","get in touch",".",{"title":376,"searchDepth":377,"depth":377,"links":378},"",2,[379,380,381,382,383,384,385,386],{"id":46,"depth":377,"text":47},{"id":62,"depth":377,"text":63},{"id":86,"depth":377,"text":87},{"id":133,"depth":377,"text":134},{"id":169,"depth":377,"text":170},{"id":267,"depth":377,"text":268},{"id":325,"depth":377,"text":326},{"id":346,"depth":377,"text":347,"children":387},[388,390,391],{"id":351,"depth":389,"text":352},3,{"id":358,"depth":389,"text":359},{"id":365,"depth":389,"text":366},"\u002Fcovers\u002Fblog\u002Fmeasuring-ai-roi-honestly.svg","Line-art illustration of a ruler measuring a stack of coins beside a rising bar.","2026-08-11","How to measure the return on an AI project without fooling yourself: baselines, full costs, holdout attribution, leading indicators and inflated-ROI red flags.",false,"md",{},true,"\u002Fblog\u002Fmeasuring-ai-roi-honestly",7,{"title":5,"description":395},"blog\u002Fmeasuring-ai-roi-honestly",[405,406],"Strategy","Analytics",null,"NIAerLO41UL48cpic9KCeleWTAMeBdd8thS1oOli3S4",[410,421,431],{"path":411,"title":412,"description":413,"date":414,"tags":415,"cover":419,"coverAlt":420,"readingTime":401},"\u002Fblog\u002Fmultimodal-ai-business-workflows","Multimodal AI in real business workflows: where it earns its place","Vision-language and audio models now read forms, photos, screenshots and calls directly. Where multimodal AI beats OCR-then-LLM, with examples, limits and cost.","2026-06-30",[416,417,418],"Computer Vision","LLMs","Automation","\u002Fcovers\u002Fblog\u002Fmultimodal-ai-business-workflows.svg","Line-art illustration of an eye, an ear and a speech bubble converging into a single processing node.",{"path":422,"title":423,"description":424,"date":425,"tags":426,"cover":429,"coverAlt":430,"readingTime":401},"\u002Fblog\u002Fmlops-basics-first-model","MLOps basics for teams shipping their first model","The minimum MLOps a team needs to put a first model into production safely: versioning, reproducible training, a registry, serving, monitoring and rollback.","2026-05-19",[427,428],"MLOps","Product","\u002Fcovers\u002Fblog\u002Fmlops-basics-first-model.svg","Line-art illustration of a looping pipeline with gauges, from data to model to monitoring and back.",{"path":432,"title":433,"description":434,"date":435,"tags":436,"cover":437,"coverAlt":438,"readingTime":401},"\u002Fblog\u002Fopen-source-llms-on-prem","Open-source LLMs on-prem: when private AI actually makes sense","Llama, Mistral and Qwen can run inside your own walls, but should they? When on-prem LLMs pay off, how to size them, and when a hosted API is the smarter call.","2026-04-07",[417,427,405],"\u002Fcovers\u002Fblog\u002Fopen-source-llms-on-prem.svg","Line-art illustration of a server rack secured with a padlock, a small model chip glowing inside.",1792973679704]