[{"data":1,"prerenderedAt":127},["ShallowReactive",2],{"work-on-prem-llm-regulated-insurer":3,"work-next-\u002Fwork\u002Fon-prem-llm-regulated-insurer":124},{"id":4,"title":5,"accent":6,"body":7,"client":83,"cover":84,"coverAlt":85,"description":86,"duration":87,"extension":88,"featured":89,"industry":90,"meta":91,"navigation":92,"order":93,"path":94,"quote":95,"results":99,"seo":109,"services":110,"stack":111,"stem":121,"year":122,"__hash__":123},"work\u002Fwork\u002Fon-prem-llm-regulated-insurer.md","On-premises open-source LLM platform for a regulated insurer","#1A4380",{"type":8,"value":9,"toc":75},"minimark",[10,20,25,28,32,59,63,68,72],[11,12,13,14,18,19],"p",{},"A German insurer wanted LLM capabilities for claims and underwriting teams but could not send policyholder data to a public API. We built an ",[15,16,17],"code",{},"open-source-ai"," platform on their own hardware (open-weight models served with vLLM on Kubernetes, retrieval over internal documents, guardrails and full observability) that now serves around 1,200 staff "," with nothing leaving the network.",[21,22,24],"h2",{"id":23},"challenge","Challenge",[11,26,27],{},"Regulators, works council and the data-protection office each had conditions: data residency inside Germany, no third-party training on prompts, role-based access and a complete audit trail. Meanwhile teams were quietly pasting documents into consumer chatbots. IT had two GPU nodes on order and no experience running inference at scale, and the board wanted a cost case against the vendor API quote already on the table.",[21,29,31],{"id":30},"what-we-built","What we built",[33,34,35,39,46,53,56],"ul",{},[36,37,38],"li",{},"A Kubernetes-based inference layer with vLLM serving quantised Llama 3.1 70B for general assistance and a smaller Mistral model for classification and extraction, behind an OpenAI-compatible gateway so internal apps could integrate without vendor lock-in.",[36,40,41,42,45],{},"A document-retrieval service (",[15,43,44],{},"rag-systems",") over policy wordings, claims guidelines and process manuals, with permission filtering tied to the insurer's identity provider via Keycloak.",[36,47,48,49,52],{},"Guardrails for prompt injection, PII redaction in logs and topic restrictions, plus prompt and response tracing in Langfuse for the audit trail (",[15,50,51],{},"mlops-ai-infrastructure",").",[36,54,55],{},"Load testing, autoscaling and GPU utilisation dashboards, with a documented runbook and two weeks of side-by-side operation with the client's platform team.",[36,57,58],{},"An evaluation harness comparing open-weight models with the API models on the insurer's own tasks before any commitment was made.",[21,60,62],{"id":61},"results","Results",[11,64,65,66,67],{},"No prompts or documents leave the network, which is the property the data-protection office signed off on. About 1,200 users were onboarded in the first quarter ",", and the measured cost per million tokens came in roughly 70% below the API quote at the insurer's volume ",", hardware amortised over three years. On the internal evaluation set the open-weight models matched the API baseline on extraction tasks and trailed slightly on long-form drafting, which the client judged an acceptable trade. ",[21,69,71],{"id":70},"what-made-the-difference","What made the difference",[11,73,74],{},"We ran the model comparison before the architecture debate. Once the insurer saw the numbers on their own tasks, the decision to go on-prem was made on evidence rather than caution, and the platform team owned it from week one.",{"title":76,"searchDepth":77,"depth":77,"links":78},"",2,[79,80,81,82],{"id":23,"depth":77,"text":24},{"id":30,"depth":77,"text":31},{"id":61,"depth":77,"text":62},{"id":70,"depth":77,"text":71},"Insurance group · life & health (Germany)","\u002Fcovers\u002Fwork\u002Fon-prem-llm-regulated-insurer.svg","Abstract navy cover with a rising cyan course line","A private LLM stack (open-weight models, GPU serving, guardrails and observability) deployed inside the client's data centre so no policy data leaves their network.","16 weeks","md",false,"Finance & Fintech",{},true,15,"\u002Fwork\u002Fon-prem-llm-regulated-insurer",{"text":96,"name":97,"role":98},"Our data-protection office signed off because they could see every component and where every byte went. That was the whole point.","Head of IT Architecture","Client (attribution on request)",[100,103,106],{"value":101,"label":102},"0","documents or prompts leaving the network",{"value":104,"label":105},"~1,200","internal users onboarded in the first quarter",{"value":107,"label":108},"~70%","lower cost per 1M tokens vs. the API quote",{"title":5,"description":86},[17,51,44],[112,113,114,115,116,117,118,119,120],"Llama 3.1 70B","Mistral","vLLM","Kubernetes","NVIDIA A100","Hugging Face","Weaviate","Langfuse","Keycloak","work\u002Fon-prem-llm-regulated-insurer","2025","X-w4lFd7coT3yjBA7kmtztrRTyooDRrxyHleqrWddD4",{"path":125,"title":126},"\u002Fwork\u002Flegal-contract-extraction-fine-tuning","Fine-tuned LLM for clause extraction from commercial contracts",1792973684198]