Notes

3 notes

Topics: attention inference llm quantization scaling-laws systems training

Jul 12, 2026 training / scaling-laws / llm

Chinchilla: The 20-Tokens-per-Parameter Rule and What Survived Its Replication

Jordan Hoffmann et al., 2022 — NeurIPS 2022

Chinchilla showed that 2022-era LLMs were dramatically undertrained, replacing 'scale parameters' with 'scale parameters and tokens together.' A 2024 replication attempt found real problems in one of its three analyses — and the headline rule survived anyway.

repro: partial

Jul 05, 2026 quantization / inference / llm

GPTQ: One-Shot 3–4 Bit Quantization as Approximate Second-Order Optimization

Elias Frantar et al., 2022 — ICLR 2023

GPTQ quantizes a 175B model to 3–4 bits in a few GPU-hours by turning layer-wise quantization into a sequence of cheap Hessian-guided weight updates — and by noticing that the expensive part of the classic algorithm was never necessary.

repro: reproduced

Jun 28, 2026 attention / inference / systems

FlashAttention-2: Where the Second 2× Actually Comes From

Tri Dao, 2023 — arXiv 2307.08691

FlashAttention-1 made attention IO-aware; FlashAttention-2 gets another ~2× by fixing how the work is divided — fewer non-matmul FLOPs, parallelism over sequence length, and warp-level partitioning that stays out of shared memory.

repro: reproduced