Worked example · Evaluation
Fine-tuning a small model for Swiss contract law.
A second worked example — same method as the FINMA report, a different domain: answering questions about the Swiss Code of Obligations (OR) with a citation to the exact article. Including a real mistake we hit on the way, and how the ablation caught it.
The setup
Four systems, one open model.
Same Apertus-8B, quantized identically to 4-bit, served identically. The only variable is the weights. Scored deterministically — no LLM judge.
Base
The open model, as shipped.
+ Retrieval
Base model with BM25 retrieval over the statute.
Fine-tuned
LoRA-tuned on the task, no retrieval.
Fine-tuned + retrieval
The full system.
The production bar is both at once — the answer correct and the correct article cited. A right answer with a wrong or missing citation is a failure, because in legal work it is one.
The result
Fine-tuning plus retrieval is the best system.
172 human-verified questions (98 German, 74 French, 10 contract-law topics). Brackets are 95% confidence intervals.
| Metric | A · base | B · +RAG | C · tuned | D · tuned+RAG |
|---|---|---|---|---|
| Answer + citation (production bar) | 4.1% | 38.4% | 2.9% | 52.3% |
| Answer correct | 39.0% | 54.7% | 46.5% | 70.3% |
| Citation correct | 7.0% | 69.8% | 5.8% | 62.2% |
| Emits the required format | 1.2% | 0.0% | 89.5% | 70.9% |
The mistake worth publishing
We didn't get 52% on the first try.
The first fine-tune was trained the obvious way — on questions and their answer + citation, without retrieved context. On its own it looked great. With retrieval added, it collapsed.
Q. Is a prior agreement excluding liability for gross negligence valid? (gold: no, Art. 100)
ANSWER: nein
Right answer, no citation → fails
ANSWER: nein SOURCE: OR Art. 100
Passes
The cause is a training/inference distribution mismatch: a model that will be served with retrieval must be trained with retrieval. We rebuilt the fine-tuning set so every example appears both plain and with retrieved context — the gold article guaranteed present — retrained, and the production bar went from 8.1% to 52.3%. That is exactly the kind of failure a Pilot's ablation catches before it reaches production. We measure so we don't ship it.
The honest limits
What this doesn't claim.
An evaluation you can trust is one that names where it's weak.
List-answer questions score near zero, in every arm. Extracting a complete multi-item set and citing it is the hardest sub-task here and the obvious next target — the same weak spot the FINMA report flagged.
French trails German (D: 42% FR vs 60% DE), tracking the training mix (271 DE vs 123 FR items). More balanced data closes it; the number is honest about where the model is weak today.
Format compliance is 71%, not 100%. The retrieval-aware fine-tune still drops the citation on some items — far better than the naive 16%, not perfect. It's a floor: a larger, more balanced training set moves it further.
FAQ
About this report.
Is this the same method as the FINMA report?
Why publish a mistake you made?
How is this scored — is there an LLM judge?
Want this on your task?
Bring one workflow and the documents behind it. We'll build the evaluation set and run this ablation — including the failure modes it catches — on your data, in your jurisdiction.