Work & case studies · Finance & Fintech
On-premises open-source LLM platform for a regulated insurer
A private LLM stack (open-weight models, GPU serving, guardrails and observability) deployed inside the client's data centre so no policy data leaves their network.
A German insurer wanted LLM capabilities for claims and underwriting teams but could not send policyholder data to a public API. We built an open-source-ai platform on their own hardware (open-weight models served with vLLM on Kubernetes, retrieval over internal documents, guardrails and full observability) that now serves around 1,200 staff with nothing leaving the network.
Challenge
Regulators, works council and the data-protection office each had conditions: data residency inside Germany, no third-party training on prompts, role-based access and a complete audit trail. Meanwhile teams were quietly pasting documents into consumer chatbots. IT had two GPU nodes on order and no experience running inference at scale, and the board wanted a cost case against the vendor API quote already on the table.
What we built
- A Kubernetes-based inference layer with vLLM serving quantised Llama 3.1 70B for general assistance and a smaller Mistral model for classification and extraction, behind an OpenAI-compatible gateway so internal apps could integrate without vendor lock-in.
- A document-retrieval service (
rag-systems) over policy wordings, claims guidelines and process manuals, with permission filtering tied to the insurer's identity provider via Keycloak. - Guardrails for prompt injection, PII redaction in logs and topic restrictions, plus prompt and response tracing in Langfuse for the audit trail (
mlops-ai-infrastructure). - Load testing, autoscaling and GPU utilisation dashboards, with a documented runbook and two weeks of side-by-side operation with the client's platform team.
- An evaluation harness comparing open-weight models with the API models on the insurer's own tasks before any commitment was made.
Results
No prompts or documents leave the network, which is the property the data-protection office signed off on. About 1,200 users were onboarded in the first quarter , and the measured cost per million tokens came in roughly 70% below the API quote at the insurer's volume , hardware amortised over three years. On the internal evaluation set the open-weight models matched the API baseline on extraction tasks and trailed slightly on long-form drafting, which the client judged an acceptable trade.
What made the difference
We ran the model comparison before the architecture debate. Once the insurer saw the numbers on their own tasks, the decision to go on-prem was made on evidence rather than caution, and the platform team owned it from week one.
Our data-protection office signed off because they could see every component and where every byte went. That was the whole point.
Next case study
Fine-tuned LLM for clause extraction from commercial contracts
Ready to chart your AI course?
A free 30-minute consultation: we listen, ask hard questions and tell you honestly where AI will and won't help.