Flash Attention
FlashAttention computes exact attention with tiling that keeps softmax stats in SRAM, cutting HBM traffic and unlocking longer contexts at the same FLOP count.
Definition
Dao et al. reformulate attention as a fused kernel: load QKV tiles, compute softmax incrementally, never materialize the full N×N matrix in HBM. Numerically it is the same softmax attention, just IO-aware.
Why it matters
This is why 8k–128k context stopped being a science project on a single GPU. If your train is attention-bound, the kernel matters more than a new positional paper.
Case
Replacing a naive PyTorch attention with FlashAttention-2 can double throughput at 4k context. The model definition did not change.
How to read it
Confirm it is exact attention, not a sparse approximation. Check alignment with your mask (causal, sliding window, padding). A wrong mask in a fused kernel is a silent correctness bug.
Key papers
Landmark references with real publisher or arXiv links — not token-overlap noise.
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessDao, Fu, Ermon, Rudra, Ré · 2022 · NeurIPSFlashAttention.