Open-source LLMs on-prem: when private AI actually makes sense

Running an open-source LLM on your own hardware makes sense when data cannot leave your environment, when volume is high enough that per-token pricing outgrows the cost of a GPU box, or when you need latency and customisation a shared API cannot give you. It does not make sense when volume is low, when your task genuinely needs frontier-model quality, or when nobody on the team can operate GPUs. Most organisations land on a hybrid: a private model for the sensitive, high-volume path and a hosted API for everything else.

TL;DR

• Go on-prem for data residency, predictable cost at scale, tight latency or deep customisation, not for novelty.
• Stay on hosted APIs if you handle under a few million tokens a day, need the best possible reasoning, or have no ops capacity.
• Size hardware from the model's weights: roughly 2 GB of VRAM per billion parameters at 16-bit, halved at 8-bit, quartered at 4-bit, plus headroom for context.
• Serve with vLLM or TGI, quantise carefully, and evaluate against the hosted model you are replacing before you commit.
• Hybrid routing gets you most of the benefit with far less risk.

When does a private LLM make sense?

Four situations come up repeatedly in our open-source AI engagements:

• Data residency and regulation. Banks, healthcare providers, public bodies and law firms often cannot send customer records to a third-party endpoint, whatever the vendor's contract says. A model in your own data centre or VPC removes the question entirely.
• Predictable cost at volume. Per-token pricing is wonderful at low volume and painful at high volume. Once you are classifying, summarising or extracting across millions of documents a month, a fixed-cost GPU server that runs flat out is usually cheaper, and the invoice stops surprising the finance team.
• Latency. A model sitting on the same network as your application answers in tens of milliseconds for the first token. That matters for voice agents, interactive tools and anything on a factory floor.
• Customisation. Fine-tuning, custom tokenisers, constrained decoding, speculative decoding, logit access: open weights give you levers a hosted API keeps behind the counter. See our note on fine-tuning for when that is worth it.

When should you stay on a hosted API?

Be honest about these before buying hardware:

• Low or bursty volume. If your workload is a few thousand requests a day, a GPU sits idle most of the time and the hosted API wins on every measure.
• You need frontier quality. Open models have closed much of the gap on structured tasks (extraction, classification, RAG answering, summarisation), but for hard multi-step reasoning, the best hosted models still lead. Test your actual task before assuming either way.
• No ops team. GPUs fail, drivers break, CUDA versions drift. If nobody owns that, on-prem becomes a science project. Our MLOps and infrastructure team exists precisely because this part is unglamorous.
• You are still exploring. Prototype on an API. Move only the workloads that survive.

How much hardware do you need?

Rules of thumb that hold up in practice:

• VRAM for weights: about 2 GB per billion parameters at FP16/BF16. An 8B model needs roughly 16 GB; a 70B model roughly 140 GB, so multiple GPUs.
• Quantisation cuts that: 8-bit halves it, 4-bit (AWQ, GPTQ, GGUF) quarters it. A 70B model at 4-bit fits in about 40 GB, comfortably on two 24 GB cards or one 80 GB card.
• Add headroom for KV cache: long contexts and many concurrent users eat memory quickly. Budget 20–40% extra above the weights, more if you promise 32k+ context to dozens of simultaneous sessions.
• Throughput: with continuous batching, a single 80 GB GPU serving an 8B model can sustain thousands of tokens per second aggregate. A 70B model on the same card at 4-bit will be several times slower per token but still viable for batch workloads.

Quantisation is not free. Expect a small quality drop at 8-bit and a noticeable one at 4-bit on reasoning-heavy tasks; measure it on your evaluation set rather than trusting the model card.

How do you serve it in production?

Two serving stacks dominate, and both are good:

• vLLM: PagedAttention, continuous batching, OpenAI-compatible endpoints, wide model support. Our default for most deployments.
• Text Generation Inference (TGI): mature, well-integrated with the Hugging Face ecosystem, straightforward to containerise.

Either way, put the model behind an OpenAI-compatible gateway so application code does not care where the model lives. Add rate limiting, per-team keys, request logging (with redaction), health checks and a canary route for new model versions. Treat model weights as versioned artefacts, not files someone copied onto a server.

Hosted API vs on-prem: how do they compare?

How do you evaluate before committing?

Never switch on faith. Build a 200–500 example evaluation set from real traffic with approved outputs, run the hosted model and two or three candidate open models over it, and score on the metric that matters: accuracy, faithfulness, format compliance, latency, cost per thousand requests. In a typical engagement, we find one open model that matches the hosted baseline on 80–90% of tasks and one or two task types where it clearly does not. That gap becomes your routing rule.

Hybrid routing is where most teams should end up: a small router (a classifier or a cheap model) sends sensitive or routine requests to the private model and escalates the hard, rare ones to a hosted API, with data redaction if policy requires it. You get residency and cost control where it counts, and frontier quality where it is worth paying for. Our generative AI and LLM practice sets these up regularly; it is rarely more than a few weeks of work once the evaluation set exists.

Key takeaways

• On-prem LLMs pay off for regulated data, sustained high volume, low latency and deep customisation.
• Hosted APIs still win at low volume, for the hardest reasoning tasks, and when no one owns the infrastructure.
• Size from parameters: about 2 GB of VRAM per billion at 16-bit, with quantisation trading memory for a measurable quality cost.
• Serve with vLLM or TGI behind an OpenAI-compatible gateway, and version your weights like code.
• Evaluate against your real traffic first, then route hybrid rather than choosing one side forever.

FAQ

Which open-source model should we start with?

Start with a mid-sized instruction-tuned model (an 8B to 14B class model from the Llama, Mistral or Qwen families), because it fits on a single GPU, serves fast and handles most extraction, classification and RAG tasks well. Only step up to 70B-class models if your evaluation set shows the smaller model failing. Licences differ between families, so have someone read the licence before you commit to a commercial deployment.

Is on-prem cheaper than a hosted API?

Only at volume. A single well-utilised GPU server, whether purchased or reserved in the cloud, has a fixed monthly cost; per-token pricing scales with usage. As an illustration, a team processing tens of millions of tokens a day usually crosses the break-even point within months, while a team at a few hundred thousand tokens a day may never reach it. Include engineering and on-call time in the comparison, not just hardware.

Do we lose quality by moving away from a hosted model?

On structured, well-scoped tasks such as extraction, summarisation, classification and grounded question answering, a fine-tuned or well-prompted open model often matches the hosted baseline. On open-ended reasoning and long multi-step tasks, hosted frontier models still tend to lead. The only reliable answer is your own evaluation set. If you would like help building one and running the comparison, get in touch.