Concepts · Evaluation
If we can't measure it, we don't ship it.
Every AI vendor promises quality. We define it with you as a number, on your documents, before we build — and gate go-live on it. This page explains the machinery behind that promise.
A reproducible four-way ablation of Apertus-8B on Swiss FINMA circular Q&A — the exact evaluation described below, run on public data.
Step one
The evaluation set comes before the model.
Built from your documents, with your experts, in your languages — it's the contract the system is built against.
We sit with your domain experts and extract what "done correctly" looks like: real inputs from your workflow — contracts, claims, notes, letters — paired with gold answers and graded on criteria you sign off. Typically 50–300 tasks across your document types, languages (DE/FR/IT as needed) and known edge cases, including the ugly ones: scanned faxes, mixed-language files, the exceptions your team handles on autopilot.
This set is deliberately built before modelling starts, so it can't be quietly tuned toward what the model happens to do well. It stays yours — versioned, documented, and reusable against every future model or vendor, including our replacement.
Step two
The four-way ablation.
One table that kills hype in both directions — ours included.
Base model
The open model, unmodified. The honest floor — and occasionally a surprise: if this already clears the bar, you shouldn't pay for fine-tuning.
+ Retrieval
Base model grounded in your corpus. Shows what better access to your documents buys — usually the biggest single jump.
+ Fine-tune
Model trained on your task without retrieval. Shows what learned behaviour buys: format, terminology, tone, refusal discipline.
Both
The production candidate. If D isn't clearly better than B and C, the report says so and you buy the cheaper system.
On the FINMA worked example, that table reads 0 → 22.4 → 1.6 → 45.4% on the production bar across A–D: retrieval alone gets a fifth of the way, fine-tuning alone almost nowhere, and only the combination doubles retrieval. The same ablation covers model choice — Apertus vs. Llama vs. Mistral on your set, and full-precision vs. quantized builds — so every architectural decision in your system traces to a measured difference, not a preference. (Why we're strict about what fine-tuning can and can't do: small models, measured.)
Step three
Metrics with teeth, guardrails with tests.
"It feels good" is not a metric. Neither is a benchmark score from someone else's task.
Task metrics match the work: exact-match accuracy for extracted fields, rubric scores for drafting, answer accuracy plus citation faithfulness for Q&A — a correct answer with a fabricated source counts as a failure, because in regulated work it is one.
Guardrails are tested like features: PII handling, out-of-scope refusals, prompt-injection resistance and output constraints each get adversarial test cases in the eval set. A guardrail that isn't tested is a wish.
The gate: the bar is agreed in the proposal. Above it, go-live. Below it, a documented gap analysis and a decision — close it, or stop. Either way you know before production money is spent.
In production
Evaluation doesn't end at go-live.
Models drift, documents change, users find the gaps. The eval set keeps watching.
Continuous scoring
Production samples are scored against the same metrics on a fixed cadence, so quality regressions surface as numbers on a dashboard — not as complaints three months later.
Drift & data change
New document types, new policy wording, new edge cases get added to the eval set as they appear. The set grows with your reality; the system is re-validated against it.
Versioned everything
Eval sets, model builds and their scores are versioned together. Every logged answer traces to the model build and eval version that produced it — for as long as your retention policy keeps the log. That trace is what your auditors will ask for.
FAQ
Evaluation questions.
How big does an evaluation set need to be?
Who defines what "correct" means?
What happens if the system misses the bar?
Define the bar with us.
Bring one workflow and the experts who know it. We'll shape the evaluation set and the metrics in the scoping call.