A technical notebook on machine learning systems
Model Foundations turns papers, source code, and benchmark tables into structured technical notes — methods worked through equation by equation, claims checked against their evidence, results placed beside their baselines, and open questions stated plainly.
Jordan Hoffmann et al., 2022 — NeurIPS 2022
Chinchilla showed that 2022-era LLMs were dramatically undertrained, replacing 'scale parameters' with 'scale parameters and tokens together.' A 2024 replication attempt found real problems in one of its three analyses — and the headline rule survived anyway.
repro: partialElias Frantar et al., 2022 — ICLR 2023
GPTQ quantizes a 175B model to 3–4 bits in a few GPU-hours by turning layer-wise quantization into a sequence of cheap Hessian-guided weight updates — and by noticing that the expensive part of the classic algorithm was never necessary.
repro: reproducedTri Dao, 2023 — arXiv 2307.08691
FlashAttention-1 made attention IO-aware; FlashAttention-2 gets another ~2× by fixing how the work is divided — fewer non-matmul FLOPs, parallelism over sequence length, and warp-level partitioning that stays out of shared memory.
repro: reproducedWhat actually matters in the work, in a few sentences.
The mechanism explained from first principles, with equations.
Each headline claim traced to the experiment behind it.
Results next to their baselines, with the setup that produced them.
What the paper does not show, and open questions.
What happened when we (or others) ran it.