Quantization gets pitched as a free lunch — smaller model, faster inference, "minimal quality loss" — and the reality is more specific than that framing suggests. The quality cost is real, it's uneven across tasks, and the right amount of quantization depends entirely on what you're using the model for. Treating it as a single dial from "full precision" to "aggressively quantized" hides the decisions that actually matter.

What's actually being reduced

Model weights are normally stored as 16-bit floating point numbers (fp16 or bf16). Quantization reduces the numeric precision used to store and compute with those weights — commonly to 8-bit integers (int8) or 4-bit representations — which directly shrinks memory footprint and, because inference is largely memory-bandwidth bound, speeds up generation by reducing how much data has to move through memory per token.

text
fp16:  16 bits/weight  →  baseline size, baseline quality
int8:   8 bits/weight  →  ~50% size, small quality loss on most tasks
int4:   4 bits/weight  →  ~25% size, noticeable loss on some tasks

The size reduction is straightforward arithmetic. The quality impact is where it gets task-dependent.

Where quality loss shows up first

Aggregate benchmark scores (perplexity, general QA accuracy) tend to hold up reasonably well down to int8 and often even int4 for many models. But aggregate scores hide where the damage actually concentrates: tasks requiring precise numeric reasoning, multi-step logical chains, and low-frequency factual recall degrade disproportionately compared to fluent text generation or general conversation. A quantized model can sound just as coherent while being measurably worse at arithmetic or at correctly recalling a specific fact it would have gotten right at full precision. If your application leans on those capabilities, benchmark on your own task — not a general leaderboard — before deciding a quantization level is safe.

Quantization methods aren't interchangeable

  • Post-training quantization (PTQ): quantize an already-trained model's weights, no retraining needed. Fast and cheap to apply, but the model wasn't trained with the numeric precision loss in mind, so quality degradation is more pronounced at aggressive bit widths.
  • Quantization-aware training (QAT): simulate quantization effects during training or fine-tuning, so the model adapts its weights to be robust to the precision reduction. Better quality at a given bit width, but requires training infrastructure and data — not a quick post-hoc step.
  • Weight-only vs. weight-and-activation quantization: quantizing only the stored weights (while computing in higher precision) is safer and simpler; quantizing activations too gives further speed gains but is more sensitive to outlier values in practice and needs more careful calibration to avoid quality cliffs.
python
# Conceptual: weight-only int8 quantization
def quantize_weights(weights_fp16, scale=None):
    scale = scale or (weights_fp16.abs().max() / 127)
    weights_int8 = (weights_fp16 / scale).round().clamp(-127, 127).to(int8)
    return weights_int8, scale  # scale needed to dequantize during compute

Deciding how far to push it

Use case Reasonable starting point
High-stakes reasoning, numeric/financial tasks fp16 or int8, validate carefully before going lower
General chat, summarization, creative tasks int8 usually safe, int4 often acceptable
Edge/on-device deployment, cost-constrained batch jobs int4 or lower, accept the trade-off deliberately
Anything user-facing with brand risk from visible errors Benchmark on your specific eval set before committing

The only reliable way to answer "how far can we quantize" is to run your own eval suite — the same one you'd use for a model swap — at each candidate precision level and compare against your specific task distribution, not a general benchmark. A quantization level that's fine for a summarization feature can be unacceptable for a feature that does arithmetic on user data, even though it's the same underlying model.

The infrastructure trade-off is real too

Lower precision isn't free even on the infrastructure side — not every GPU has efficient kernels for every bit width, and int4 inference sometimes trades memory savings for compute overhead depending on hardware support. Validate the actual latency and throughput gain on your target hardware rather than assuming linear scaling with bit width; the theoretical size reduction doesn't always translate into a proportional speed gain in practice.

A worked failure mode

A team int8-quantizes a model that writes invoices. BLEU on a chat eval is "close enough," but amounts like 1099 become 1098 in a handful of cases. They ship because mean exact-match on a general benchmark barely moved. A second team 4-bit quantizes embeddings used for legal search; nearest neighbors shuffle enough that a critical statute falls out of the top five. The failure is using a generic quality number for a numeric or retrieval-sensitive task. Keep a task-specific exact-match slice, compare calibration methods, and leave residual layers in higher precision if that is where the errors concentrate. Quantization is a product change, not a compiler flag.

When this is the wrong tool

If the model already fits with margin and latency is fine, do not quantize for the blog post. If errors on digits, code, or rare names are unacceptable and you lack evals, quantization is the wrong knob. Do not mix quantized embeddings with an index built from full-precision vectors without rebuilding. Training-aware quantization is overkill for a one-off batch job. Prefer a smaller full-precision model you can eval over an aggressive 3-bit experiment on the critical path.