Concepts · Quantization
How a datacentre model fits in your office.
Quantization is the unglamorous technology that decides whether sovereign AI costs a server or a datacentre. Here's how it actually works, in enough detail for your engineers to check.
The mechanism
A model's size is bytes per weight × weights.
Training uses 16-bit numbers. Inference doesn't have to — most of that precision is redundancy you can spend down.
Quantization stores each weight in fewer bits — grouped and rescaled so the model's behaviour is preserved. Half the bits, half the memory, and usually faster too: at the low-batch, single-tenant loads on-prem systems actually run, generation speed is limited by how fast weights stream from memory, so a smaller model is a quicker one.
| Precision | 8B model | 70B model | Runs on |
|---|---|---|---|
| 16-bit (training) | ~16 GB | ~140 GB | Multi-GPU node |
| 8-bit | ~8 GB | ~70 GB | Datacentre GPU(s) |
| 4-bit (the sweet spot) | ~4.5 GB | ~40 GB | One workstation GPU / one 48 GB card (tight — or 2×24 GB) |
| 1–2 bit (the frontier) | ~1.5–2.5 GB | ~12–20 GB | Laptop or edge box (a phone, for 27B-class and below) |
Weights only — working memory adds the KV cache (below) and runtime overhead. The 70B-on-one-48-GB-card fit assumes a compact 4-bit variant and a quantized cache; exact figures depend on format and grouping, and we size the full budget during scoping.
The trade
What it costs, and how we keep it honest.
Quantization trades a measurable sliver of quality for a 3–4× cut in memory and the hardware tier below it. The operative word is measurable.
4-bit: the production default
Modern 4-bit formats — GGUF K-quants for llama.cpp-style serving, AWQ or GPTQ elsewhere — typically cost a point or two on broad benchmarks; on a narrow, fine-tuned task, often nothing detectable. This is where most sovereign deployments should live: a 70B in ~40 GB of weights, an 8B in under 5 GB.
Below 4 bits: engineered, not naive
Simply rounding harder destroys quality. Purpose-built ternary and binary formats — increasingly paired with quantization-aware training — keep models useful at 1–2 bits per weight, shrinking a 27B-class model to phone size. This frontier is moving fast, and it's the one we watch most closely.
Our rule: re-evaluate, then ship
Every quantized build is re-run against your evaluation set and shipped with the quality delta documented next to the full-precision baseline. If quantization costs more than the deployment gain is worth for your task, the report says so and we size up.
The second budget
Context has a memory bill too.
Weights are only half the sizing exercise. The other half is the KV cache — the model's working memory of your conversation or document.
For every token in context, the model stores key-value activations. At short contexts this is noise; at 100K-token contexts — a long contract, a full case file — the cache grows into the gigabytes and can rival the weights themselves.
The cache quantizes too: 4-bit KV storage cuts its footprint several-fold, which is often the difference between "long documents fit" and "they don't" on a given machine. Architecture matters as well — newer models with hybrid attention keep the cache disproportionately small for their size.
Practical consequence: when we commit to "handles your 200-page contracts on one machine", that claim has been sized — weights, cache, overhead — not hoped.
Why it matters here
Quantization is sovereignty's cost curve.
Every bit shaved off a weight moves "runs where your data lives" one step down the hardware ladder.
At 16-bit, running your own model is a capital request with a floor plan attached. At 4-bit, it's one server. At the 1–2 bit frontier, it starts meaning laptops, edge boxes and machines that have never seen a network cable — the environments where regulated data actually lives.
That's why we treat quantization as a first-class discipline, not a checkbox. See what the resulting systems run on in the sovereign runtime.
FAQ
Quantization questions.
Does quantization make the model worse?
Why does long context need so much memory?
Want the sizing for your use case?
Tell us the documents and the hardware you have — we'll come back with the memory budget and the machine it runs on.