Worked example · Healthcare & Pharma
Does fine-tuning help on Swiss therapeutic-products regulation?
Not an opinion — a measurement. We took Apertus-8B and ran the same four-way ablation we run at the start of every Pilot, on a task where the answer can be scored exactly: answering questions about the Swiss Therapeutic Products Act with a citation to the exact article it rests on — with a citation to the exact article it rests on. The result: fine-tuning plus retrieval takes the production bar from 22.7% to 68.2%.
The setup
Four systems, one open model.
Same Apertus-8B, quantized identically to 4-bit, served identically. The only variable is what we added.
Base
The open model, as shipped.
+ Retrieval
Base model with BM25 retrieval over the law.
Fine-tuned
LoRA-tuned on the task, no retrieval.
Fine-tuned + retrieval
The full system.
Two things are scored, both deterministically — no LLM judge: is the answer correct, and did it cite the correct article. The production bar is both at once — a right answer with a wrong or missing citation is a failure, because in regulated work it is one.
The result
Fine-tuning plus retrieval is the winning system.
22 verified questions (14 German, 8 French). Brackets in the report are 95% confidence intervals.
| Metric | A · base | B · +RAG | C · tuned | D · tuned+RAG |
|---|---|---|---|---|
| Answer + citation (production bar) | 4.5% | 22.7% | 0.0% | 68.2% |
| Answer correct | 54.5% | 81.8% | 59.1% | 81.8% |
| Citation correct | 13.6% | 27.3% | 4.5% | 72.7% |
| Emits the required format | 0% | 0% | 100% | 100% |
What each layer buys
The ablation stops anyone — us included — from hand-waving.
Each card pins one mechanism to a number in the table above.
Retrieval enables citation
Citation-correct climbs from 13.6% to 27.3% only when retrieval is on. A small model cannot cite an article number it has never seen — so both no-retrieval arms sit near zero on the production bar, however well-trained.
Fine-tuning enables usable output
The base model often knows the answer (81.8% with RAG) but emits the required, machine-parseable two-line format just 0% of the time. The fine-tune makes it reliable (100%), and lifts citation under retrieval to 72.7%.
Fine-tuning alone is not enough
Arm C — fine-tuned, no retrieval — reaches 100% format but only 4.5% citation and 59.1% answer: the model learns how to answer, not the exact facts. The winning system is the combination, arm D at 68.2%.
The honest limits
What this doesn't claim.
An evaluation you can trust names where it's weak.
68.2% is a floor, not a ceiling. One small model, 4-bit, a single modest LoRA run, deliberately plain BM25 retrieval, a compact verified eval set (n=22). Every one of those is a lever a real Pilot pulls.
Fine-tuning alone can regress. Arm C's answer accuracy sits below the base model — proof that fine-tuning without retrieval is the wrong tool for exact-citation tasks. We publish that rather than hide it.
The method is the product, not the number. Your task, your documents, this exact ablation — run before you commit to production, so the go/no-go rests on measurement, not a promise.
FAQ
About this report.
Is this on our data?
Why does fine-tuning alone (arm C) fail?
Is there an LLM judge?
Want this on your task?
Bring one workflow and the documents behind it. We'll build the evaluation set and run this ablation — on your data, in your jurisdiction.