Worked example · Evaluation
Does fine-tuning a small model actually help?
Not an opinion — a measurement. We took Apertus-8B and ran the same four-way ablation we run at the start of every Pilot, on a task where the answer can be scored exactly: answering questions about Swiss FINMA circulars with a citation to the exact margin number it rests on. The result: fine-tuning plus retrieval doubles the production bar — 22.4% to 45.4%.
The setup
Four systems, one open model.
Same Apertus-8B, quantized identically to 4-bit, served identically. The only variable is what we added.
Base
The open model, as shipped.
+ Retrieval
Base model with BM25 retrieval over the circulars.
Fine-tuned
LoRA-tuned on the task, no retrieval.
Fine-tuned + retrieval
The full system.
Two things are scored, both deterministically — no LLM judge: is the answer correct, and did it cite the correct Rz in the correct circular. The production bar is both at once — a right answer with a wrong or missing citation is a failure, because in regulated work it is one.
The result
Fine-tuning plus retrieval doubles the production bar.
183 human-verified questions (101 German, 82 English, 10 circulars). Brackets are 95% confidence intervals.
| Metric | A · base | B · +RAG | C · tuned | D · tuned+RAG |
|---|---|---|---|---|
| Answer + citation (production bar) | 0.0% | 22.4% | 1.6% | 45.4% |
| Answer correct | 37.2% | 63.4% | 44.8% | 80.9% |
| Citation correct | 1.1% | 39.9% | 2.2% | 50.3% |
| Emits the required format | 2.2% | 0.0% | 100% | 100% |
What each layer buys
The ablation stops anyone — us included — from hand-waving.
Each card below pins one mechanism to a number in the table above.
Retrieval enables citation
Citation-correct goes from ~1% to ~40–50% only when retrieval is on. A model cannot cite a margin number it has never seen — so both no-retrieval arms sit near zero on the production bar, however well-trained.
Fine-tuning enables usable output
The base model often knows the answer (63% with RAG) but emits the required, machine-parseable format just 2% of the time — 0% once retrieval fills the context and distracts it. The fine-tune nails the format 100% of the time. That gap is most of B → D.
The two compound
Retrieval alone adds 22.4 points; fine-tuning alone adds 1.6. Together they add 45.4 — nearly double the sum of the parts. That interaction is the finding, and it's why we measure before we build.
It looks like this
Same question, same retrieved text.
The base model isn't stupid — it's unreliable in shape. Reliability of shape is what separates a demo from a system.
Q. For banks, securities firms and financial groups, what is the minimum required equity capital at which the fully qualitative and quantitative requirements apply? (gold: CHF 10 billion, Rz 6, FINMA-RS 2010/01)
CHF 10 billion
Right answer, no citation → fails
ANSWER: CHF 10 billion SOURCE: FINMA-RS 2010/01 Rz 6
Passes
The honest limits
What this doesn't claim.
An evaluation you can trust is one that names where it's weak.
45% is a floor, not a ceiling. One small model, 4-bit, a single modest LoRA run on ~500 examples, deliberately plain BM25 retrieval. Every one of those is a lever a real Pilot pulls.
Structured answers are the weak spot. Fine-tuned+RAG scores ~47–50% on numbers, durations and yes/no but 27% on multi-item list questions. Complete-set extraction is the obvious next target.
The method is the product, not the number. Your task, your documents, this exact ablation — run before you commit to production, so the go/no-go decision rests on measurement, not a promise.
FAQ
About this report.
Is this on your customers' data?
Why is 45% the headline and not something higher?
How is this scored — is there an LLM judge?
Want this on your task?
Bring one workflow and the documents behind it. We'll build the evaluation set and run this ablation — on your data, in your jurisdiction.