Quantization, Explained: From FP32 to INT4, and What Each Precision Costs You

When you run a modern language model, memory is not just an implementation detail — it defines what is physically possible. A 27-billion parameter model stored in full 32-bit floating point takes up over 108 GB of VRAM just for its weights, immediately exceeding the physical capacity of an 80 GB GPU like the NVIDIA H100. If you want to serve it on a single card, or serve it with high throughput and reasonable cost, you have to compress those numbers.

That is where quantization comes in: the practice of storing weights and activations in fewer bits than the precision they were trained on. Done carelessly, quantization quietly degrades reasoning ability or turns text generation into gibberish. Done well, it shrinks a 108 GB model down to 13.5 GB with almost zero noticeable loss in quality.

In this guide, we walk through how quantization actually works under the hood — starting from the binary anatomy of floating-point numbers, exploring why hardware designers introduced specialized formats like TF32 and BF16, and examining how modern post-training algorithms (GPTQ, AWQ, and the NF4 format behind QLoRA) keep 4-bit models remarkably faithful to their 16-bit counterparts, backed by published benchmarks.

1. What a floating-point number actually is

At its core, a floating-point number is binary scientific notation: a sign, an exponent that shifts the scale, and a mantissa (or fraction) that supplies the digits within that scale.

Here $m$ represents the number of mantissa bits. Every floating-point format is a zero-sum trade-off between range and precision:

• More exponent bits widen the range: the span between the largest and smallest magnitudes the format can represent before overflowing to infinity or underflowing to zero.
• More mantissa bits sharpen precision: the spacing between adjacent representable numbers is tighter, preserving fine-grained mathematical differences.

Within a fixed bit budget, you cannot maximize both. Every format on the market represents a deliberate compromise tailored to a specific hardware or training workload.

Two crucial takeaways emerge when you look at these bit layouts side by side:

1. TF32 is not a storage format: Although it only uses 19 meaningful bits (1 sign, 8 exponent, 10 mantissa), NVIDIA pads it to a standard 32-bit container in memory. It saves zero VRAM over standard FP32 at rest; its sole purpose is speeding up Tensor Core matrix multiplies during execution.
2. BF16 and FP16 reflect opposing philosophies: Both occupy exactly 16 bits, but they allocate them completely differently. FP16 prioritizes precision (10 mantissa bits) at the expense of exponent range (5 bits). BF16 (Brain Floating Point) does the reverse: it preserves FP32’s full 8-bit dynamic range, sacrificing precision (only 7 mantissa bits). That single architectural choice explains why BF16 took over deep-learning training.

2. Why training moved to BF16, and what TF32 actually is

During backpropagation in a deep transformer, gradients and activations swing wildly across orders of magnitude. FP16’s 5-bit exponent gives it a dynamic range capped at roughly 65,504. In large models, useful gradient updates frequently exceed that ceiling, overflowing directly to inf and corrupting training with NaN losses. Smaller gradients vanish into underflow. To prevent crashes, engineers had to use loss-scaling heuristics that dynamically rescaled values to squeeze them into FP16’s narrow window.

BF16 eliminated that headache entirely. By matching FP32’s 8 exponent bits, BF16 provides an identical dynamic range (up to $\approx 3.4 \times 10^{38}$). Values that would catastrophically overflow FP16 simply round to a slightly less precise BF16 neighbor. Because gradient descent is remarkably tolerant of rounded mantissas but completely brittle to blown exponents, BF16 rapidly became the universal default for training frontier models.

TF32 (TensorFloat-32) solves a different problem: accelerating FP32 code without requiring developers to rewrite pipelines. Introduced with NVIDIA’s Ampere architecture (A100), TF32 is not a format you store on disk, but an execution mode inside Tensor Cores. Your tensors remain standard 32-bit floats in memory. When Tensor Cores execute a matrix multiply, the hardware reads the inputs, truncates the mantissa from 23 bits down to 10 bits internally, performs the multiply at Tensor Core speeds, and accumulates the result in full FP32.

The result is the best of both worlds for legacy software: FP32’s full exponent range, FP16-like math speed, and zero code changes. In fact, PyTorch enables TF32 by default on Ampere and newer GPUs. If you trained an “FP32” baseline on modern hardware, you were almost certainly running TF32 under the hood.

3. From floats to integers: what quantization does

Unlike floating-point numbers, integer formats have no exponent bits at all. An INT8 tensor is simply a uniform grid counting from −128 to 127 (or 0 to 255 unsigned), where every single step is the exact same size.

To map continuous floating-point weights onto discrete integer buckets, quantization introduces two external values:

• A scale factor ($s$): a floating-point multiplier setting how wide each integer step is in real units.
• A zero-point ($z$): an integer offset aligning the real value 0.0 with a specific integer index.

Here $x$ is the original floating-point value, $q$ is the stored quantized integer, and $\hat{x}$ is the dequantized value reconstructed during math.

In practice, quantizers use two primary configurations:

• Symmetric quantization ($z=0$): When weights are centered symmetrically around zero (typical for neural network layers), we drop the zero-point offset entirely and map values into [−127, 127] based on the maximum absolute magnitude. This saves computation because the hardware does not need to subtract an offset during matrix multiplies.
• Asymmetric quantization: Retains both scale and zero-point. This is essential for activations that are skewed or strictly positive, such as outputs after ReLU or SiLU functions.

The other key architectural choice is granularity:

• Per-tensor: One single scale and zero-point for the entire weight matrix. Highly compact, but easily ruined if a few values in the tensor are unusually large.
• Per-channel (or per-row): A dedicated scale for each output channel. Standard for INT8 weight quantization.
• Per-group: Dividing each row into small blocks of 32, 64, or 128 weights, giving each block its own scale. This is the foundation of modern 4-bit algorithms like GPTQ and AWQ: it adds a modest metadata footprint (storing scales per group), but prevents local spikes from degrading unrelated weights.

4. Model size at each precision

Calculating weight memory is straightforward: total parameter count multiplied by bytes per parameter. While runtime memory also demands space for KV-cache, activations, and workspace buffers, the raw weights establish the baseline footprint required just to load the model.

Notice how these thresholds line up with real-world hardware:

• At FP32 (108 GB), the weights alone exceed the physical capacity of an 80 GB NVIDIA H100 or A100. Running the model requires multi-GPU sharding.
• Dropping to BF16 (54 GB) fits the weights onto a single 80 GB card, but leaves constrained memory for deep concurrency and long context windows.
• At INT4 (13.5 GB), the entire model fits comfortably on a single consumer GPU (like an RTX 4090 with 24 GB) or leaves more than 65 GB of free VRAM on an H100 for high-throughput KV-cache batching. This same math underpins QLoRA: a frozen 4-bit base model consumes only 13.5 GB, allowing adapters to train efficiently in the remaining space.

5. The outlier problem

If quantizing a tensor were as simple as laying an even ruler between its minimum and maximum values, low-bit quantization would have been solved years ago. The complication is a peculiar mathematical phenomenon that emerges in deep language models.

In 2022, Dettmers et al. discovered that once transformer models grow past roughly 6.7 billion parameters, their internal representations undergo a phase transition: they develop emergent outlier features. Across specific hidden dimensions, activation magnitudes suddenly spike to values 50× to 100× larger than all other channels.

Look at what happens in Panel B: because a single outlier is 12× larger than its peers, the quantizer is forced to stretch its scale factor ($s$) by 12× to avoid clipping that outlier to infinity.

As a result, all the normal, everyday weights collapse onto just one or two integer buckets near zero. All fine-grained precision is obliterated. This “outlier catastrophe” is why early naive per-tensor INT8 and INT4 implementations suffered steep drops in accuracy, and why modern post-training algorithms were designed specifically to isolate and protect these extreme values.

6. Post-training quantization in practice

Modern quantization algorithms solve the outlier problem using four distinct strategies:

• LLM.int8() (Mixed-Precision Decomposition): Rather than forcing outliers into INT8, Dettmers et al. (2022) isolates the ~0.1% outlier coordinate dimensions and keeps them in full 16-bit floating point. The remaining 99.9% of normal values are multiplied in INT8, and the results are summed. By exempting the extreme channels from quantization entirely, it eliminates outlier distortion without loss of accuracy.
• GPTQ (Second-Order Error Compensation): Frantar et al. (2022) quantizes weights layer by layer. When a weight is rounded to an integer, it introduces a small rounding error. Instead of ignoring that error, GPTQ uses second-order (Hessian) curvature information to immediately adjust the remaining unquantized weights in that row to compensate. Because it dynamically balances rounding errors, GPTQ achieves stable 4-bit compression in a few hours without retraining, typically operating at per-group granularity (128 weights per scale).
• AWQ (Activation-Aware Weight Quantization): Lin et al. (2023) recognized that not all weights are equally important. Weights that multiply high-magnitude activations have an outsized impact on output perplexity, whereas weights multiplying near-zero activations can be rounded aggressively. Rather than keeping outliers in FP16 (which complicates runtime kernels), AWQ scales up salient weight channels before quantization and scales the corresponding activation inputs down. Protecting just the top 1% most salient channels preserves model quality under clean, uniform INT4 execution.
• NF4 (NormalFloat-4 in QLoRA): Dettmers et al. (2023) observed that standard INT4 grids assume weights are uniformly distributed. But neural network weights cluster in a bell curve (normal distribution) around zero. NF4 creates an information-theoretically optimal grid where all 16 discrete levels represent equal quantiles of a standard normal distribution. More quantization bins cluster near zero where most weights live, with wider spacing covering the rare tails. Paired with double quantization (quantizing the block scale constants themselves into 8-bit), NF4 achieves near-lossless 4-bit compression for QLoRA fine-tuning.

7. What the published benchmarks show

The empirical literature confirms that naive rounding fails at low bit-widths, while structured quantizers preserve baseline performance:

The lesson across every benchmark is consistent: naive round-to-nearest (RTN) degrades quickly because it treats every parameter as uniform noise. But as soon as an algorithm accounts for error propagation (GPTQ), activation salience (AWQ), empirical weight distributions (NF4), or dedicated outlier channels (LLM.int8()), models retain virtually all of their native capability — even compressed down to 4 bits.

8. Picking a precision for your workload

There is no single “best” precision. The optimal choice depends on whether you are training, optimizing for latency, or packing maximum batch density:

One final operational reality: benchmark perplexity does not tell the whole story. A 4-bit model that looks flawless on a WikiText perplexity test can still regress on delicate edge cases — such as multi-step mathematical proofs, strict JSON schema output, or niche code syntax. Always validate your quantized checkpoint directly on domain verifiers and task-specific gyms before shipping to production.