Fine-tuned LLM for clause extraction from commercial contracts

A contract-lifecycle software company was using a prompted frontier model to extract clauses from customer contracts. Accuracy plateaued and per-document costs were eating margin. Through our ai-fine-tuning practice we fine-tuned an 8B open-weight model with LoRA on the client's own annotated contracts, lifting field-level F1 from about 0.91 to 0.96 while cutting inference cost by roughly 85%.

Challenge

The product extracted more than forty clause types (governing law, termination triggers, indemnity caps, auto-renewal terms) from contracts in inconsistent formats. Prompt engineering had squeezed out most of the gains available; edge cases such as nested exceptions and cross-references kept failing, and every extra example in the prompt increased cost. The client had thousands of reviewed contracts but no clean training set, and no in-house ML engineers.

What we built

A data programme first: we converted lawyer-reviewed extractions from the product's own history into a training set of about 6,000 contracts , with a Label Studio workflow to resolve disagreements and a held-out test set that never touched training.
LoRA fine-tuning of Llama 3.1 8B with instruction-formatted examples, tuned for structured JSON output and calibrated abstention on clauses that are absent.
An evaluation harness scoring per-field precision and recall, error slices by contract type and length, and side-by-side comparison with the prompted frontier baseline on every run.
Serving with vLLM on the client's AWS account, an A/B rollout behind their existing API, and a retraining pipeline triggered when reviewers correct enough new examples (open-source-ai).
Documentation and a training session so the client's engineers can rerun the pipeline themselves.

Results

Field-level F1 rose from around 0.91 to 0.96 on the held-out set , with the largest gains on the nested and cross-referenced clauses that had caused the most support tickets. Cost per contract dropped by roughly 85% , and median latency fell to a level that let the client show extractions live during upload rather than by email later. Related pillar: nlp for the pre-processing and section segmentation.

What made the difference

Most of the effort, and most of the gain, went into the dataset, not the training run. Cleaning labels, agreeing definitions with the client's lawyers and building a trustworthy test set turned fine-tuning from an experiment into a repeatable process the client now runs without us.