[{"data":1,"prerenderedAt":413},["ShallowReactive",2],{"blog-open-source-llms-on-prem":3,"blog-related-\u002Fblog\u002Fopen-source-llms-on-prem":384},{"id":4,"title":5,"author":6,"authorRole":7,"body":8,"cover":366,"coverAlt":367,"date":368,"description":369,"draft":370,"extension":371,"meta":372,"navigation":373,"path":374,"readingTime":375,"seo":376,"stem":377,"tags":378,"updated":382,"__hash__":383},"blog\u002Fblog\u002Fopen-source-llms-on-prem.md","Open-source LLMs on-prem: when private AI actually makes sense","William Ali","Founder & AI Specialist",{"type":9,"value":10,"toc":349},"minimark",[11,15,43,48,57,88,92,95,126,130,133,159,162,166,169,183,186,190,279,283,286,297,301,318,322,327,330,334,337,341],[12,13,14],"p",{},"Running an open-source LLM on your own hardware makes sense when data cannot leave your environment, when volume is high enough that per-token pricing outgrows the cost of a GPU box, or when you need latency and customisation a shared API cannot give you. It does not make sense when volume is low, when your task genuinely needs frontier-model quality, or when nobody on the team can operate GPUs. Most organisations land on a hybrid: a private model for the sensitive, high-volume path and a hosted API for everything else.",[16,17,18,24],"blockquote",{},[12,19,20],{},[21,22,23],"strong",{},"TL;DR",[25,26,27,31,34,37,40],"ul",{},[28,29,30],"li",{},"Go on-prem for data residency, predictable cost at scale, tight latency or deep customisation, not for novelty.",[28,32,33],{},"Stay on hosted APIs if you handle under a few million tokens a day, need the best possible reasoning, or have no ops capacity.",[28,35,36],{},"Size hardware from the model's weights: roughly 2 GB of VRAM per billion parameters at 16-bit, halved at 8-bit, quartered at 4-bit, plus headroom for context.",[28,38,39],{},"Serve with vLLM or TGI, quantise carefully, and evaluate against the hosted model you are replacing before you commit.",[28,41,42],{},"Hybrid routing gets you most of the benefit with far less risk.",[44,45,47],"h2",{"id":46},"when-does-a-private-llm-make-sense","When does a private LLM make sense?",[12,49,50,51,56],{},"Four situations come up repeatedly in our ",[52,53,55],"a",{"href":54},"\u002Fservices\u002Fopen-source-ai","open-source AI"," engagements:",[25,58,59,65,71,77],{},[28,60,61,64],{},[21,62,63],{},"Data residency and regulation."," Banks, healthcare providers, public bodies and law firms often cannot send customer records to a third-party endpoint, whatever the vendor's contract says. A model in your own data centre or VPC removes the question entirely.",[28,66,67,70],{},[21,68,69],{},"Predictable cost at volume."," Per-token pricing is wonderful at low volume and painful at high volume. Once you are classifying, summarising or extracting across millions of documents a month, a fixed-cost GPU server that runs flat out is usually cheaper, and the invoice stops surprising the finance team.",[28,72,73,76],{},[21,74,75],{},"Latency."," A model sitting on the same network as your application answers in tens of milliseconds for the first token. That matters for voice agents, interactive tools and anything on a factory floor.",[28,78,79,82,83,87],{},[21,80,81],{},"Customisation."," Fine-tuning, custom tokenisers, constrained decoding, speculative decoding, logit access: open weights give you levers a hosted API keeps behind the counter. See our note on ",[52,84,86],{"href":85},"\u002Fservices\u002Fai-fine-tuning","fine-tuning"," for when that is worth it.",[44,89,91],{"id":90},"when-should-you-stay-on-a-hosted-api","When should you stay on a hosted API?",[12,93,94],{},"Be honest about these before buying hardware:",[25,96,97,103,109,120],{},[28,98,99,102],{},[21,100,101],{},"Low or bursty volume."," If your workload is a few thousand requests a day, a GPU sits idle most of the time and the hosted API wins on every measure.",[28,104,105,108],{},[21,106,107],{},"You need frontier quality."," Open models have closed much of the gap on structured tasks (extraction, classification, RAG answering, summarisation), but for hard multi-step reasoning, the best hosted models still lead. Test your actual task before assuming either way.",[28,110,111,114,115,119],{},[21,112,113],{},"No ops team."," GPUs fail, drivers break, CUDA versions drift. If nobody owns that, on-prem becomes a science project. Our ",[52,116,118],{"href":117},"\u002Fservices\u002Fmlops-ai-infrastructure","MLOps and infrastructure"," team exists precisely because this part is unglamorous.",[28,121,122,125],{},[21,123,124],{},"You are still exploring."," Prototype on an API. Move only the workloads that survive.",[44,127,129],{"id":128},"how-much-hardware-do-you-need","How much hardware do you need?",[12,131,132],{},"Rules of thumb that hold up in practice:",[25,134,135,141,147,153],{},[28,136,137,140],{},[21,138,139],{},"VRAM for weights:"," about 2 GB per billion parameters at FP16\u002FBF16. An 8B model needs roughly 16 GB; a 70B model roughly 140 GB, so multiple GPUs.",[28,142,143,146],{},[21,144,145],{},"Quantisation cuts that:"," 8-bit halves it, 4-bit (AWQ, GPTQ, GGUF) quarters it. A 70B model at 4-bit fits in about 40 GB, comfortably on two 24 GB cards or one 80 GB card.",[28,148,149,152],{},[21,150,151],{},"Add headroom for KV cache:"," long contexts and many concurrent users eat memory quickly. Budget 20–40% extra above the weights, more if you promise 32k+ context to dozens of simultaneous sessions.",[28,154,155,158],{},[21,156,157],{},"Throughput:"," with continuous batching, a single 80 GB GPU serving an 8B model can sustain thousands of tokens per second aggregate. A 70B model on the same card at 4-bit will be several times slower per token but still viable for batch workloads.",[12,160,161],{},"Quantisation is not free. Expect a small quality drop at 8-bit and a noticeable one at 4-bit on reasoning-heavy tasks; measure it on your evaluation set rather than trusting the model card.",[44,163,165],{"id":164},"how-do-you-serve-it-in-production","How do you serve it in production?",[12,167,168],{},"Two serving stacks dominate, and both are good:",[25,170,171,177],{},[28,172,173,176],{},[21,174,175],{},"vLLM:"," PagedAttention, continuous batching, OpenAI-compatible endpoints, wide model support. Our default for most deployments.",[28,178,179,182],{},[21,180,181],{},"Text Generation Inference (TGI):"," mature, well-integrated with the Hugging Face ecosystem, straightforward to containerise.",[12,184,185],{},"Either way, put the model behind an OpenAI-compatible gateway so application code does not care where the model lives. Add rate limiting, per-team keys, request logging (with redaction), health checks and a canary route for new model versions. Treat model weights as versioned artefacts, not files someone copied onto a server.",[44,187,189],{"id":188},"hosted-api-vs-on-prem-how-do-they-compare","Hosted API vs on-prem: how do they compare?",[191,192,193,209],"table",{},[194,195,196],"thead",{},[197,198,199,203,206],"tr",{},[200,201,202],"th",{},"Dimension",[200,204,205],{},"Hosted API",[200,207,208],{},"On-prem open-source",[210,211,212,224,235,246,257,268],"tbody",{},[197,213,214,218,221],{},[215,216,217],"td",{},"Upfront cost",[215,219,220],{},"None",[215,222,223],{},"GPUs, or reserved cloud instances, plus setup",[197,225,226,229,232],{},[215,227,228],{},"Cost at high volume",[215,230,231],{},"Grows linearly with tokens",[215,233,234],{},"Roughly flat once hardware is utilised",[197,236,237,240,243],{},[215,238,239],{},"Data control",[215,241,242],{},"Contractual",[215,244,245],{},"Physical and technical",[197,247,248,251,254],{},[215,249,250],{},"Peak model quality",[215,252,253],{},"Best available",[215,255,256],{},"Strong on structured tasks, behind on hardest reasoning",[197,258,259,262,265],{},[215,260,261],{},"Latency",[215,263,264],{},"Network + queue dependent",[215,266,267],{},"Local, predictable",[197,269,270,273,276],{},[215,271,272],{},"Ops burden",[215,274,275],{},"Minimal",[215,277,278],{},"Real: drivers, upgrades, capacity, on-call",[44,280,282],{"id":281},"how-do-you-evaluate-before-committing","How do you evaluate before committing?",[12,284,285],{},"Never switch on faith. Build a 200–500 example evaluation set from real traffic with approved outputs, run the hosted model and two or three candidate open models over it, and score on the metric that matters: accuracy, faithfulness, format compliance, latency, cost per thousand requests. In a typical engagement, we find one open model that matches the hosted baseline on 80–90% of tasks and one or two task types where it clearly does not. That gap becomes your routing rule.",[12,287,288,291,292,296],{},[21,289,290],{},"Hybrid routing"," is where most teams should end up: a small router (a classifier or a cheap model) sends sensitive or routine requests to the private model and escalates the hard, rare ones to a hosted API, with data redaction if policy requires it. You get residency and cost control where it counts, and frontier quality where it is worth paying for. Our ",[52,293,295],{"href":294},"\u002Fservices\u002Fgenerative-ai-llm","generative AI and LLM"," practice sets these up regularly; it is rarely more than a few weeks of work once the evaluation set exists.",[44,298,300],{"id":299},"key-takeaways","Key takeaways",[25,302,303,306,309,312,315],{},[28,304,305],{},"On-prem LLMs pay off for regulated data, sustained high volume, low latency and deep customisation.",[28,307,308],{},"Hosted APIs still win at low volume, for the hardest reasoning tasks, and when no one owns the infrastructure.",[28,310,311],{},"Size from parameters: about 2 GB of VRAM per billion at 16-bit, with quantisation trading memory for a measurable quality cost.",[28,313,314],{},"Serve with vLLM or TGI behind an OpenAI-compatible gateway, and version your weights like code.",[28,316,317],{},"Evaluate against your real traffic first, then route hybrid rather than choosing one side forever.",[44,319,321],{"id":320},"faq","FAQ",[323,324,326],"h3",{"id":325},"which-open-source-model-should-we-start-with","Which open-source model should we start with?",[12,328,329],{},"Start with a mid-sized instruction-tuned model (an 8B to 14B class model from the Llama, Mistral or Qwen families), because it fits on a single GPU, serves fast and handles most extraction, classification and RAG tasks well. Only step up to 70B-class models if your evaluation set shows the smaller model failing. Licences differ between families, so have someone read the licence before you commit to a commercial deployment.",[323,331,333],{"id":332},"is-on-prem-cheaper-than-a-hosted-api","Is on-prem cheaper than a hosted API?",[12,335,336],{},"Only at volume. A single well-utilised GPU server, whether purchased or reserved in the cloud, has a fixed monthly cost; per-token pricing scales with usage. As an illustration, a team processing tens of millions of tokens a day usually crosses the break-even point within months, while a team at a few hundred thousand tokens a day may never reach it. Include engineering and on-call time in the comparison, not just hardware.",[323,338,340],{"id":339},"do-we-lose-quality-by-moving-away-from-a-hosted-model","Do we lose quality by moving away from a hosted model?",[12,342,343,344,348],{},"On structured, well-scoped tasks such as extraction, summarisation, classification and grounded question answering, a fine-tuned or well-prompted open model often matches the hosted baseline. On open-ended reasoning and long multi-step tasks, hosted frontier models still tend to lead. The only reliable answer is your own evaluation set. If you would like help building one and running the comparison, ",[52,345,347],{"href":346},"\u002Fcontact","get in touch",".",{"title":350,"searchDepth":351,"depth":351,"links":352},"",2,[353,354,355,356,357,358,359,360],{"id":46,"depth":351,"text":47},{"id":90,"depth":351,"text":91},{"id":128,"depth":351,"text":129},{"id":164,"depth":351,"text":165},{"id":188,"depth":351,"text":189},{"id":281,"depth":351,"text":282},{"id":299,"depth":351,"text":300},{"id":320,"depth":351,"text":321,"children":361},[362,364,365],{"id":325,"depth":363,"text":326},3,{"id":332,"depth":363,"text":333},{"id":339,"depth":363,"text":340},"\u002Fcovers\u002Fblog\u002Fopen-source-llms-on-prem.svg","Line-art illustration of a server rack secured with a padlock, a small model chip glowing inside.","2026-04-07","Llama, Mistral and Qwen can run inside your own walls, but should they? When on-prem LLMs pay off, how to size them, and when a hosted API is the smarter call.",false,"md",{},true,"\u002Fblog\u002Fopen-source-llms-on-prem",7,{"title":5,"description":369},"blog\u002Fopen-source-llms-on-prem",[379,380,381],"LLMs","MLOps","Strategy",null,"9P2RGLwtQatjtWmh5xO0lXgHxSUNMOO3Nc0HOJb1iAM",[385,394,404],{"path":386,"title":387,"description":388,"date":389,"tags":390,"cover":392,"coverAlt":393,"readingTime":375},"\u002Fblog\u002Fmeasuring-ai-roi-honestly","Measuring AI ROI honestly: a method that survives the finance review","How to measure the return on an AI project without fooling yourself: baselines, full costs, holdout attribution, leading indicators and inflated-ROI red flags.","2026-08-11",[381,391],"Analytics","\u002Fcovers\u002Fblog\u002Fmeasuring-ai-roi-honestly.svg","Line-art illustration of a ruler measuring a stack of coins beside a rising bar.",{"path":395,"title":396,"description":397,"date":398,"tags":399,"cover":402,"coverAlt":403,"readingTime":375},"\u002Fblog\u002Fmultimodal-ai-business-workflows","Multimodal AI in real business workflows: where it earns its place","Vision-language and audio models now read forms, photos, screenshots and calls directly. Where multimodal AI beats OCR-then-LLM, with examples, limits and cost.","2026-06-30",[400,379,401],"Computer Vision","Automation","\u002Fcovers\u002Fblog\u002Fmultimodal-ai-business-workflows.svg","Line-art illustration of an eye, an ear and a speech bubble converging into a single processing node.",{"path":405,"title":406,"description":407,"date":408,"tags":409,"cover":411,"coverAlt":412,"readingTime":375},"\u002Fblog\u002Fmlops-basics-first-model","MLOps basics for teams shipping their first model","The minimum MLOps a team needs to put a first model into production safely: versioning, reproducible training, a registry, serving, monitoring and rollback.","2026-05-19",[380,410],"Product","\u002Fcovers\u002Fblog\u002Fmlops-basics-first-model.svg","Line-art illustration of a looping pipeline with gauges, from data to model to monitoring and back.",1792973683660]