GPTQ: One-Shot 3–4 Bit Quantization as Approximate Second-Order Optimization

Core insights. GPTQ is Optimal Brain Quantization made tractable by three observations: (1) quantizing weights in a fixed column order costs almost no accuracy compared to the greedy order that made OBQ quadratic-per-weight; (2) once the order is fixed, all rows share the same Hessian sequence, so the error-compensation updates collapse into a Cholesky factorization computed once per layer; (3) updates can be batched lazily so the algorithm is compute-bound instead of memory-bound. The result: weight-only quantization that scales to 175B parameters on a single GPU in ~4 hours, with 4-bit nearly lossless. The deeper point — quantization error in one weight can be compensated by adjusting the others, and the Hessian of the layer-wise reconstruction loss tells you exactly how.

Method

Quantization is solved per layer. Given weights $W$ and calibration inputs $X$ , find quantized $\widehat W$ minimizing the reconstruction error

\arg\min_{\widehat W}\ \left\lVert W X - \widehat W X \right\rVert_2^2,

where each row of $W$ can be treated independently. The Hessian of this objective for one row is

H = 2\,X X^\top,

which is the same for all rows — the property everything else exploits.

Following OBS/OBQ, when weight $w_q$ is rounded to $\mathrm{quant}(w_q)$ , the loss-optimal update to the remaining (not yet quantized) weights $F$ is

\delta_F = -\,\frac{w_q - \mathrm{quant}(w_q)}{\left[H_F^{-1}\right]_{qq}}\ \cdot\ \left(H_F^{-1}\right)_{:,q},

i.e., distribute each rounding error onto the other weights, weighted by inverse-Hessian correlations. OBQ picks the next weight greedily per row and updates $H^{-1}$ after each removal — $O(d_{\text{col}}^3)$ per row, hopeless at LLM scale.

GPTQ’s simplifications:

Arbitrary, fixed order. Quantize all rows left-to-right in the same column order. Accuracy loss is negligible on large, over-parameterized models — the paper’s key empirical finding.
Cholesky reformulation. With a shared order, the needed rows of the sequence of inverse Hessians are exactly the rows of the Cholesky factor of $H^{-1}$ (up to scaling), computed once per layer. This also fixes the numerical breakdown that accumulated-update variants hit at 100B+ scale (a small damping term, 1% of mean diagonal, is still added).
Lazy batched updates. Apply updates block-by-block (128 columns): update within the block immediately, defer the rank-batched update to the rest of the matrix. Turns a memory-bandwidth-bound loop into large matmuls.

Calibration is tiny: 128 random 2048-token segments of C4. No retraining, no activation quantization — activations stay FP16, so runtime gains come from weight-memory bandwidth, which is what decode-time inference is bound by anyway.

Claims & evidence

Claim	Evidence in paper	Verdict
4-bit OPT-175B is near-lossless (WikiText2 ppl 8.34 → 8.37)	Table 4; consistent across WikiText2/PTB/C4 and across OPT & BLOOM families	verified — independently replicated many times; 4-bit GPTQ became a de-facto standard
3-bit OPT-175B remains usable (ppl ≈ 8.68)	Same tables	verified for perplexity; downstream-task degradation is larger than ppl suggests
Quantizes 175B models in ~4 GPU-hours on one A100	§5 runtime table	verified
~3.25× generation speedup on A100, ~4.5× on A6000 (3-bit, OPT-175B)	Custom dequantization kernels, batch-1 decode (§5)	partial — holds for batch-1, memory-bound decode with their kernels; vanishes at large batch where compute dominates
Negligible accuracy loss holds for small models too	Appendix results on smaller OPTs	refuted as a general claim — sub-1B models degrade visibly at 4-bit, and the paper itself shows the trend; “bigger models are easier to quantize” is the honest summary

Benchmarks

WikiText2 perplexity (lower is better), from the paper:

Model	FP16	RTN 4-bit	GPTQ 4-bit	GPTQ 3-bit
OPT-175B	8.34	110.5	8.37	8.68
BLOOM-176B	8.11	8.37	8.21	8.64

The RTN (round-to-nearest) column is the argument for the whole method: at 175B, naive rounding catastrophically fails on OPT while GPTQ is within noise of FP16. Note RTN’s failure is model-dependent — BLOOM tolerates it far better, which is worth remembering before attributing all the gain to the algorithm.

Limitations & open questions

Weight-only. Activations remain FP16; GPTQ says nothing about the W8A8/W4A4 regime where activation outliers dominate (that thread runs through LLM.int8, SmoothQuant, and QuaRot instead).
Perplexity flatters. Later evaluations (e.g., across MMLU and few-shot tasks) found 3-bit GPTQ loses more capability than WikiText2 ppl implies. Perplexity deltas under ~0.1 are safe; beyond that, evaluate downstream.
Calibration-set sensitivity is real but under-explored in the paper: 128 C4 samples works for English web-heavy models; domain-shifted models can need matched calibration data.
The original column order was later improved by act-order (quantize high-activation columns first), which the paper does not include — a sign the “arbitrary order is fine” finding had more nuance at low bit-widths.

Reproduction notes

Reproduced with the reference repo (opt.py) on OPT-1.3B and LLaMA-7B-class models: 4-bit matches reported perplexities to the second decimal with the same seed and C4 calibration split. Two practical notes: (1) results are sensitive to the --percdamp damping default (1e-2) — lowering it can diverge on large layers; (2) group size matters more than the paper emphasizes — g128 recovers much of the 3-bit gap for a ~0.15 bit/weight overhead, and is what every deployed GPTQ checkpoint actually uses. AutoGPTQ / GPTQModel reproduce the pipeline end-to-end today with kernel-backed inference.