Counting the bits a low-bit model actually stores

The label names the grid, not the storage

Start concrete. Take a 4096-weight output channel quantized to 2 bits with group size 128, the setting behind EfficientQAT’s numbers. Each weight is a 2-bit integer, so the codes cost exactly 2 bits per weight. But every group of 128 weights also needs its own scale, the step size of that group’s grid, stored in fp16, and that scale is a real 16 bits you have to keep. Spread over the 128 weights it serves, it adds $16/128 = 0.125$ bits per weight. So the honest cost is

$$\text{eff} = b + \frac{\text{metadata bits per group}}{g} = 2 + \frac{16}{128} = 2.125 ,$$

which is the 2.12 that ParetoQ’s table credits to EfficientQAT. Shrink the group to 64 and the same 16-bit scale is amortized over half as many weights: $2 + 16/64 = 2.25$. Finer groups buy accuracy and cost bits, and the exchange rate is exactly $16/g$.

There is a second charge that papers include or drop at will. An asymmetric quantizer stores a zero point as well as a scale (the white paper’s default for activations, and what EfficientQAT trains). Count that zero point as another fp16 number and the overhead doubles to $32/g$, so w2g128 becomes $2 + 32/128 = 2.25$ under the same label that read 2.12 a moment ago. Which number a paper prints depends on whether it counts the zero point, packs it into an integer, or folds it away. I have not found a paper that states its convention in the main text; AWQ, for one, reports group-quantized bit-widths without writing the formula down. So a 0.125-bit difference between two “w2g128” rows can be a real storage difference or a bookkeeping choice, and the label will not tell you which.

The “2-bit” column is not a column

Put the uniform-grid numbers side by side and the single “2-bit” heading stops meaning one thing:

Label	Effective bits/weight
channel-wise (one scale per row)	~2.0
w2g128, scale only	2.125
w2g64, scale only	2.25
w2g128, scale + zero point	2.25

That is a 0.25-bit spread, about 12% more storage, between rows a table would stack under “2-bit” and then rank by accuracy. ParetoQ is careful about this and reports its own result as channel-wise 2.0 against EfficientQAT’s group-wise 2.12, which makes its win cost fewer bits, not more; most citations of that table drop the distinction and read 71.2 versus 65.5 as two models of the same size. They are not the same size. The ParetoQ model is about 6% smaller per weight, and that is part of the result, not a footnote to it.

The same soft-accounting problem, one axis over, sits underneath a discrepancy the EfficientQAT note tracks: ParetoQ credits EfficientQAT’s Llama-3-8B w2g128 with 65.5 while EfficientQAT’s own paper reports 59.37 for that setting. Effective bits is not the cause there, since both numbers name the same g128 config; the cause is the accuracy metric, acc_norm against plain acc, which moves that cell by six points. Same lesson, different column. Comparison tables carry axes the headline hides, group size and zero-point accounting on the storage side, evaluation metric and harness version on the accuracy side, and any one of them can move a model by more than the gap being reported.

Vector quantization counts differently again

The codebook methods (AQLM, QuIP#, QTIP) do not round to a grid at all, so “bits” means something else and the arithmetic changes. A weight is stored as an index into a learned codebook of short vectors. With $M$ codebooks of $K$ entries each over groups of $d$ weights, the indices cost $M\log_2(K)/d$ bits per weight, and the codebook itself, $K$ vectors of $d$ fp16 numbers, is stored once and amortized over every weight that uses it.

AQLM’s main 2-bit configuration is one codebook of $2^{15}$ entries over groups of 8 weights (their appendix). The indices alone are $15/8 = 1.875$ bits per weight. The codebook adds a term that genuinely depends on matrix size: for a $4096\times4096$ projection sharing one codebook it is another 0.25 bits (total 2.125), for an $11008$-wide MLP matrix only 0.035 bits (total 1.91). AQLM reports ~2.0 averaged; ParetoQ’s table lists its AQLM entry at 2.27. Same method, two accountings, and neither is wrong, they are amortizing the codebook over different denominators. The reason vector quantization can beat a uniform grid at the same nominal bits is exactly this freedom: the codebook cells sit wherever the weight distribution is dense instead of on an even ruler, which is the QuIP incoherence story run in reverse.

Reading a bit column

None of this makes low-bit tables useless; it makes the bit column data rather than a caption. Read the group size, since w2g64 and channel-wise 2.0 are a quarter-bit apart before any accuracy is measured. Ask whether a method’s “2-bit” counts its zero points and codebooks or quietly excludes them. And when a method wins by a point or two at a nominal bit-width, check that it is not spending a tenth of a bit more to do it. The methods on this site are mostly careful about their own accounting; the danger lives in the cross-paper table, where each row was measured by a different group under a different convention and the shared column header hides all of it.

What I verified

The arithmetic is a short standard-library script, verification/effective-bits/checks.py: the group-overhead formula reproducing 2.125 for w2g128 and 2.25 for both w2g64 and scale-plus-zero at g128, the 0.25-bit spread across rows all labeled “2-bit,” and AQLM’s 1.875 index cost with the codebook amortization bracketing its reported ~2.0. Every accuracy number I quote (71.2, 65.5, 59.37, 70.2) and every bit label I attribute to a paper (2.12, 2.27) is sourced to the note or paper it comes from and was not recomputed here; this note checks the storage arithmetic, not the benchmarks.