ParetoQ, the strongest low-bit QAT I could find

Method

Why this paper

The brief I set myself: the best 2–3 bit quantization-aware training result available today, judged on average benchmarks against other methods, with a paper, code, and LaTeX source I could audit. I compared the QAT lineage (LLM-QAT, BitDistiller, EfficientQAT, ParetoQ) and then went hunting for 2025–26 papers claiming to beat it. The hunt turned up one real challenger and a lot of noise. LC-QAT (ICML 2026) reports 73.42 versus ParetoQ’s 72.07 on the Llama-3-8B six-task average using 4B tokens against 30B, which is a serious efficiency claim; but its perplexity is worse (9.38 vs 8.00), it loses two of the six tasks, it stores weights as vector-quantization codebooks rather than plain integers, and the paper contains no code link anywhere despite its abstract page promising one. Bit-by-Bit (April 2026) is the boldest claim, a sweep of ParetoQ on every metric; but the fine print says its “ParetoQ” baseline was re-trained by the authors on roughly 1/3600th of the original token budget to match their own setup, and the resulting baseline numbers (45.8 zero-shot average on Llama-3.2-3B, versus ~63 in ParetoQ’s own results) are nowhere near the published ones. Read it as a data-efficiency study with a mislabeled headline, and note it too ships no code. Elsewhere: D2Quant (2026) is a PTQ method that never compares against ParetoQ and lands 21.7 points below full precision at 2-bit; R2Q (2025) doesn’t mention ParetoQ at all; pQuant publishes neither comparable numbers nor code. ParetoQ is NeurIPS-accepted, beats every auditable baseline across 1–4 bits, and ships an official Meta repo with released checkpoints. If LC-QAT’s code appears, it deserves this same treatment; until then it’s a claim, not a result.

QAT versus PTQ, if you’re coming from the GPTQ note

The GPTQ note covers post-training quantization: take a finished model, round its weights carefully, repair the damage, never train again. That machinery collapses at 2 bits. In the comparison table below, GPTQ at 2 bits scores 35.9, and random guessing on those benchmarks scores about 35. Nothing PTQ-shaped survives four allowed values per weight, because there is no longer a “nearby” grid point to repair toward.

Quantization-aware training takes the opposite bet: put the rounding inside training and let gradient descent find weights that work despite it. Concretely, the model keeps full-precision “shadow” weights. Every forward pass quantizes them on the fly, so the loss is computed with the low-bit weights the deployed model will actually have; the optimizer then updates the shadow copies. One obstacle: rounding is a staircase function whose derivative is zero almost everywhere, so gradients cannot flow through it honestly. The standard workaround, used here, is the straight-through estimator (STE): in the backward pass, pretend the rounding step was the identity function wherever the weight is inside the clipping range, and pass the gradient straight through (utils_quant.py:103). It is a lie, but a productive one, and at QAT’s scale of thousands of steps the lie averages out well enough to train 8B models.

The cost is real training: ParetoQ’s 2-bit runs use around 30B tokens. The payoff is that the network can do what no repair rule can, which is move weights far, and the paper’s most interesting content is measuring when that’s needed.

Finding 1: the 90/10 budget split

Fix a total token budget. How much should go to full-precision pretraining versus QAT? The paper sweeps the ratio on MobileLLM-125M at four bit-widths and gets the same answer everywhere: accuracy climbs slowly with more full-precision share, peaks near 90/10, then falls off a cliff at 100/0 (which is PTQ) because nothing is left for adaptation. QAT from scratch (0/100), the BitNet recipe, loses at every bit-width tested. Initialize from a good full-precision model, then adapt briefly.

The sweep is on a 125M model and the paper extrapolates the 90/10 rule upward; I’d call the exact ratio weakly supported at 8B scale, and the qualitative shape (both endpoints lose, the optimum sits near but not at the full-precision end) strongly supported.

Finding 2: 2 bits is where the model rewrites itself

The paper measures how far weights move during QAT, normalized to their scale. At 3 and 4 bits, weights shift by roughly 10–20%: the network stays near its pretrained solution and the finetuning acts as compensation, the same within-neighborhood adjustment that makes GPTQ-style error repair work. At 2 bits and below, weights move around 40%: the network is not adjusting its old representation, it is reconstructing a new one that happens to be expressible in four values per weight.

This is the finding I keep thinking about, because it gives the whole field a clean boundary: above the transition, cheap post-hoc methods are fighting for the same within-neighborhood gains as QAT; below it, they are structurally unable to follow, since no local repair reaches a reconstructed solution. It also predicts the token costs: the paper finds 3–4 bit QAT saturates after ~10B tokens while 2-bit and below need ~30B, which is what you’d expect if one regime is touching up and the other is re-learning.

Finding 3: at low bits, the grid is not a detail

Here is the part where the paper earns its “we checked everything” framing. A quantization grid for $b$ bits has $2^b$ levels, and you must choose where they sit. The standard integer grid (used by LSQ, the method 3/4-bit inherits here) puts levels at $\{-2^{b-1}, \dots, 2^{b-1}-1\}$ times a step size $\alpha$: for 4 bits, sixteen levels from $-8\alpha$ to $+7\alpha$, including zero. Two properties of that choice go unnoticed at 4 bits and become the whole game at 2. It contains zero, which costs one level; and it is asymmetric (one more negative level than positive), which is irrelevant when you have sixteen levels and painful when you have four: $\{-2, -1, 0, 1\}\alpha$ gives exactly one positive value.

ParetoQ’s proposal for 2-bit and ternary, Stretched Elastic Quant (SEQ), gives up the zero level in exchange for symmetry and even coverage. I verified the grids numerically from the code: SEQ at 2 bits produces levels $\{-0.75, -0.25, +0.25, +0.75\}\,\alpha$, four balanced values with no zero, evenly tiling the clipped range; ternary SEQ produces $\{-\tfrac{2}{3}, 0, +\tfrac{2}{3}\}\,\alpha$, keeping zero where an odd level count makes it free. The step size $\alpha$ is not a statistic but a trained parameter, one per output channel, initialized from the channel’s maximum weight (train.py:49–55) and updated by gradient descent with the LSQ scaling rule. The paper’s ablations show the crossover cleanly: SEQ wins at ternary and 2-bit, plain LSQ wins at 3 and 4.

The mechanism-level punchline connects back to Finding 2: in the reconstruction regime the network is choosing a new weight distribution, so the grid’s job is to offer the most usable set of destinations, and a balanced zero-free set beats a lopsided zero-containing one. In the compensation regime the network wants to stay near its old (roughly zero-centered, outlier-heavy) distribution, and zero plus fine steps wins.

The 2-bit recipe, in full

Since 2-bit is the setting most people will come here for, everything I could pin down about how the headline model was actually trained, in one place. Each value carries its source; where the paper is silent, I say so.

Setting	Value	Source
Initialization	pretrained FP weights; scales $\alpha = \max\lvert w\rvert$ per channel	§5; train.py:49–55
Quantized layers	all linears; embeddings and output layer stay FP	§5, confirmed in code
Quantizer	SEQ, per-output-channel learned $\alpha$, clip 0.99	utils_quant.py:151–161
Optimizer	AdamW, weight decay 0	§5
Learning rate	$2\times10^{-5}$, cosine decay to zero, no warmup in the demo script	§5; 1_run_train.sh
Iterations	120,000 (vs 40,000 for 3/4-bit)	§5
Batch	16 GPUs × 8 per GPU = 128 sequences of 2048 tokens	§5; run script
Total tokens	120k × 128 × 2048 ≈ 31.5B, matching the quoted “30B”	derived
Precision	bf16 compute, TF32 explicitly off	run script; utils_quant.py:13–14
Gradient accumulation, warmup, Adam betas	not stated for the production run	absent from paper
Data	~30B tokens, corpus never named	see below

One trap for the careful reader: the paper contains a second training configuration in its appendix (64 GPUs, learning rate $1\times10^{-4}$, linear decay) which belongs to the MobileLLM-125M budget-sweep experiments behind Finding 1, not to the headline models. Quote the right one.

The loss is plain next-token cross-entropy, and there is no teacher. The repo uses the stock HuggingFace Trainer (train.py:88), labels are a copy of the input tokens (datautils.py:112), and the model calls the default causal-LM loss (modeling_llama_quant.py:869). No distillation, no KL against the full-precision model, no reconstruction penalty, no regularizer on the scales. This is a real differentiator: LLM-QAT distills from the FP model, BitDistiller uses confidence-aware self-distillation, EfficientQAT minimizes block-wise reconstruction error, and ParetoQ beats all of them at 2-bit with the same objective used for pretraining. A leftover in the checkpoint-saving code that filters out parameters named “teacher” (utils.py:44) hints that a distillation variant existed internally; what shipped is teacher-free.

So what is the full-precision model for, if nothing distills from it? It becomes the training state. The FP weights are loaded as the trainable shadow weights that every forward pass projects onto the 2-bit grid, and nothing anchors training back toward them: no L2-to-init term, no teacher signal, just cross-entropy from the starting position. That resolves the apparent tension with Finding 2. Yes, 2-bit training rewrites weights by ~40%, but 30B tokens is around 0.2% of Llama-3’s pretraining and cannot teach language from scratch; all the competence in the final model was paid for during FP pretraining and carried through the rewrite. The from-scratch endpoint of Finding 1, which is the BitNet philosophy, loses at every bit-width for exactly this reason.

During training, the optimizer sees the quantizer through two channels. Weights inside the clipping range get the straight-through gradient (round treated as identity); weights outside get exactly zero, since the clamp is genuinely flat out there (utils_quant.py:240). Frozen weights are not dead, though: the scale $\alpha$ keeps receiving a boundary-value gradient from clipped weights, so the range can grow to re-capture them. The weight learns inside the fence, the fence learns from what it excludes.

And the data: the corpus is never named. The paper specifies optimizer, schedule, GPUs, and iteration counts, but not what text was trained on; the repo takes a user-supplied JSONL path, and the README’s only hint is that the released base models were trained on “a more advanced data”. Every quantitative claim in this note is therefore attached to the recipe plus Meta’s data, and third-party re-runs on open corpora (RedPajama substitutions, instruction-tuned models) have so far come back weaker, which is the reproduction flank discussed under Limitations.

Paper vs. code

Repo: facebookresearch/ParetoQ at commit 7b36b8d, official, BSD-3. The quantization core is models/utils_quant.py, 290 lines; everything else is a standard training harness. You don’t need to read code to use this section.

What I could match directly:

Paper	Code
SEQ for 1.58/2-bit, LSQ for 3/4-bit (Eq. 3)	the exact dispatch in QuantizeLinear.forward, utils_quant.py:266–283
SEQ formula $\alpha(\lfloor\mathrm{Clip}(w/\alpha)\cdot k/2 - 0.5\rceil + 0.5)/k \times 2$	utils_quant.py:154–159; my grid check reproduces the paper’s stated 2-bit levels
Learned per-channel scale	weight_clip_val, shape (rows, 1), utils_quant.py:259
LSQ gradient for $\alpha$ with $1/\sqrt{n \cdot Q_p}$ scaling	utils_quant.py:41–44, 76–101, matching the LSQ paper’s recipe
STE with clipping-range gate	utils_quant.py:103, 240
Scale init: $\max\lvert w\rvert$ per channel (SEQ), $\max\lvert w\rvert/(2^{b-1}-1)$ (LSQ)	train.py:49–55
Embeddings and output layer stay full-precision	stated in §5; confirmed, only nn.Linear replacements are quantized

What the code does that the paper doesn’t say:

• The paper’s Eq. 3 clips $w/\alpha$ to $[-1, 1]$. Taken literally, the boundary value $+1$ maps to $\mathrm{round}(1.5) = 2$ under round-half-even, producing a fifth level $+1.25\alpha$ that does not fit in 2 bits. The code clips to $\pm 0.99$ (clip_val = 1 - 1e-2, utils_quant.py:141), which quietly removes the overflow. I verified both behaviors numerically; the published equation and the shipped quantizer are not the same function, and the code’s version is the correct one.
• QuantizeLinear hard-disables bias (utils_quant.py:254), fine for Llama-family models which have none, silently wrong for anything with linear biases.
• The training data pipeline is a user-supplied JSONL path (utils/datautils.py:30); the 30B-token corpus behind the released checkpoints is not shipped, so exact reproduction of the headline numbers is not possible from the repo alone.
• Gradients through the quantizer are computed in FP32 against the shadow weights, and the quantized weights are cast to the activation dtype per forward (utils_quant.py:274); there is no packed-integer kernel anywhere in the repo. Training-time memory is the full-precision model plus optimizer states; the memory savings exist only after export.

Claims & evidence

Zero-shot accuracy below means the average score over commonsense-reasoning multiple-choice benchmarks (ARC, PIQA, HellaSwag, WinoGrande and similar), where random guessing lands in the mid-30s.

Claim	Evidence in paper	Verdict
2-bit Llama-3 8B within 3.4 points of full precision (71.2 vs 74.6)	Table 2, five-task average; checkpoints released on HuggingFace	verified as reported; I did not re-run their eval
Beats the best prior QAT (EfficientQAT) by 5.7 points at 2-bit	Same table: 71.2 vs 65.5	verified per the table, with two caveats: EfficientQAT’s entry is group-wise 2.12 effective bits against ParetoQ’s channel-wise 2.0, so ParetoQ wins with the smaller footprint; and both 71.2 and 65.5 are acc_norm, so they are fairly compared, whereas EfficientQAT’s own paper reports 59.37 (plain acc) for this model, a metric difference and not a disagreement (traced in the EfficientQAT note)
Outperforms the vector-quantization method AQLM by 7.1 points	§5 text says AQLM scores 64.1; the paper’s own Table 2 lists AQLM at 70.2, making the margin 1.0 point at a higher 2.27-bit budget	partial — the text and the table disagree with each other; by the table, ParetoQ still wins but narrowly, against a method spending more bits
~90% FPT / 10% QAT is the optimal budget split	Ratio sweep on MobileLLM-125M only (Fig. 3)	partial; shape is convincing, the exact ratio is extrapolated to larger models without a sweep
Ternary 600M ParetoQ beats the previous SoTA ternary 3B model	Fig. 6, against 1-bit-era checkpoints	unverified here — plausible from their figure, but I did not obtain the baseline checkpoints
2-bit QAT behavior transition (compensation vs reconstruction)	Weight-deviation analysis, Fig. 4	verified as a measurement; the mechanistic interpretation is the authors’ hypothesis, clearly labeled as such

Benchmarks

Llama-3 8B, 2-bit weight quantization. Five-task zero-shot average (acc_norm where the task defines it, higher is better) and WikiText2 perplexity (lower is better), from the paper’s Table 2, which was built from the baselines’ released models:

Method	Type	Effective bits	Avg. acc.	Wiki2 ppl
Full precision	n/a	16	74.6	6.15
RTN	PTQ	2	35.7	1.2e6
GPTQ	PTQ	2	35.9	162
SpinQuant	PTQ	2	38.1	31.2
LLM-QAT	QAT	2	54.3	29.5
AQLM	PTQ+FT	2.27	70.2	n/a
EfficientQAT	QAT	2.12	65.5	9.6
ParetoQ	QAT	2	71.2	8.0

Read it in three tiers. Pure PTQ (RTN, GPTQ, SpinQuant) is at or near chance: 2-bit is simply past what post-hoc repair can reach, which is the reconstruction regime made visible. Older QAT (LLM-QAT) survives but bleeds twenty points. The modern contenders (AQLM with vector codebooks and finetuning, EfficientQAT, ParetoQ) form the real race, and ParetoQ leads it while using the fewest effective bits of the three. One cell worth a footnote: EfficientQAT’s 65.5 here is ParetoQ’s acc_norm re-run of the released checkpoint, while EfficientQAT’s own paper reports 59.37 for the same model under plain acc. Both are right for their own tables and the gap is entirely the metric, traced in the EfficientQAT note; this whole table is acc_norm, so its rows are metric-consistent with each other.

Limitations & open questions

The cost is the story the headline hides. Thirty billion tokens of QAT for the 2-bit 8B model is a real pretraining-scale bill, thousands of GPU-hours, not the four GPU-hours of a GPTQ run. Whether that cost is worth four points over EfficientQAT (which needs ~41 GPU-hours) depends entirely on how many devices will serve the model.

Effective-bits bookkeeping muddies every low-bit comparison, this one included. ParetoQ reports channel-wise 2.0-bit numbers against baselines at 2.12 and 2.27 effective bits; that makes its win more impressive, but the table’s bit column deserves as much attention as its accuracy column, and most citations of the paper drop it.

The largest model is 8B. Nothing here says what the compensation/reconstruction boundary does at 70B, where per-parameter capacity is more redundant and PTQ methods (the AQLM/QuIP# family) historically close the gap.

The scale claims stop at accuracy: the repo contains no low-bit inference kernel, so the deployment story (2-bit is faster and smaller in practice) rests on the paper’s appendix hardware experiments, which I did not audit.

And the reproducibility gap: without the training corpus, nobody outside Meta can regenerate the headline numbers, only evaluate the released checkpoints. Two 2026 threads press on this from different sides. LC-QAT’s claim of a better task average with a tenth of the tokens would, if its code ever surfaces, change the cost calculus substantially, though its worse perplexity suggests the win is not uniform. And an under-review line of work (“Compensate, Don’t Reconstruct”, OpenReview 2026) reports that ParetoQ’s recipe degrades on instruction-tuned models when reproduced with open-source data; I could not access their numbers past OpenReview’s bot wall, but the direction of the critique is plausible, since everything in this note was measured on base models with Meta’s internal corpus.

Reproduction notes

Marked partial, meaning: on 2026-07-12 I read models/utils_quant.py and the relevant parts of train.py end to end at commit 7b36b8d and matched them against the paper’s equations, which is the Paper-vs-code section. I reimplemented both quantizer forward passes in NumPy and verified the grids the paper describes, including the boundary overflow in the published equation that the code’s 0.99 clip fixes; the script is at verification/paretoq-low-bit-qat/checks.py and every quoted level set prints as a PASS line. Benchmark numbers were transcribed from the paper’s LaTeX table source, not from screenshots.

What I did not do: re-run any training (the 2-bit recipe needs ~30B tokens and the corpus isn’t public) or re-evaluate the released HuggingFace checkpoints, which is the natural next step for anyone wanting to promote the headline claim from “verified as reported” to “reproduced”. The checkpoints are at facebook/MobileLLM-ParetoQ-* if you want to run that eval before I do.