FlashAttention-2: Where the Second 2× Actually Comes From
Tri Dao, 2023 — arXiv 2307.08691
FlashAttention-1 made attention IO-aware; FlashAttention-2 gets another ~2× by fixing how the work is divided — fewer non-matmul FLOPs, parallelism over sequence length, and warp-level partitioning that stays out of shared memory.
repro: reproduced