Standard attention materializes an N × N score matrix in GPU high-bandwidth memory. For a 4k context that matrix is already huge; for 16k it starts to dominate memory and the kernel spends its time waiting on loads, not computing fused multiply-adds. FlashAttention's claim is blunt: the naive algorithm is IO-bound. If you fuse softmax and never write the full matrix to HBM, you get exact attention (not an approximation) that is faster and uses less memory.

That is why long-context training became plausible on the same chips, and why almost every serious training stack now ships a FlashAttention-shaped kernel (or a vendor equivalent) rather than a naive PyTorch matmul + softmax + matmul.

Tiling, online softmax, and exactness

The kernel loads blocks of Q, K, and V into on-chip SRAM, computes a block of scores, and updates the output incrementally using an online softmax that tracks running maxima and sums. Because softmax is numerically sensitive to the max, the algorithm has to be careful — it is not "just loop in tiles." The paper's contribution is showing you can do this without changing the mathematical result.

Approximate attention (Linformer, Performer, sparse patterns) tried to dodge the quadratic cost by changing the function. FlashAttention kept the function and changed the memory traffic. For many production sequence lengths, that was the better trade: you still pay quadratic compute, but you stop paying quadratic HBM writes.

What to take into a design review

  • If someone proposes "we cannot do 32k context, attention is O(n²) FLOPs," ask whether they are actually OOM on the score matrix first. Memory is often the binding constraint before FLOPs.
  • If someone proposes a new approximate attention for a 2k coding model, ask for a kernel-level profile. You may already be compute-bound, and approximation would only hurt quality.
  • Training and inference have different shapes. Inference decode is often memory-bound on the KV cache, which is a related but not identical IO story (FlashDecoding, paged attention, GQA). Do not cite FlashAttention as if it solved decode.

The paper is a reminder that "the algorithm" and "the algorithm as it hits the memory hierarchy" are different objects. In GPU-era ML, the second object is frequently the one that decides whether the first one can ship.

A worked example

Profile naive attention vs FlashAttention on a 8k sequence, batch 2, hidden 1024. NVIDIA Nsight or a simple CUDA memory snapshot shows the naive path allocating a multi-gigabyte scores tensor; FlashAttention keeps activations near the expected O(N) range for that layer. Training step time drops even though FLOP count is similar. You record both step time and peak allocated bytes in the experiment log so a later "optimization" that reintroduces materialization is obvious.

When enabling a vendor kernel, lock versions: a silent fallback to math attention on an unsupported head dimension looks like a performance regression in the app, not a kernel miss.

Failure modes

Assuming FlashAttention changed complexity class — it is still quadratic compute. Mixed precision softmax numerical edge cases on extreme score ranges. Custom masks that the kernel does not support falling back slowly. Training with sequence packing where padding tokens still participate because the mask was not passed into the fused kernel. Inference decode bottlenecks on KV cache bandwidth while someone "turns on FlashAttention" and expects decode tokens/s to double.

Compiling for one GPU arch and running on another silently uses a worse kernel.

When this is the wrong tool

Tiny sequences (hundreds of tokens) may already be compute-bound; fusion wins less. Approximate attention may still be required at 1M context if you cannot pay quadratic FLOPs. FlashAttention does not replace model architecture choices (GQA, sliding window) for memory at extreme length. CPU inference and tiny edge NPUs need different kernels. If you cannot measure HBM traffic, do not argue from the paper title alone — measure bytes and time.

Review checklist

  • OOM vs FLOPs is measured on the score matrix before proposing approximations.
  • Masks and packing are actually consumed by the fused kernel.
  • Decode/KV-cache IO is a separate discussion from training attention.
  • Kernel arch matches the GPU you run; fallbacks are visible in profiles.

A worked failure mode

An inference stack enables a new attention kernel on GPUs that do not have the required tensor-core path, falls back silently to a memory-heavy implementation, and OOMs at context 8k after a "successful" deploy. Another team copies a fused kernel into a training job with custom masking and gets wrong gradients on padded tokens because the mask was not plumbed. Loss still decreases; a few sequence positions are systematically ignored. The failure is treating FlashAttention as a compiler switch with no golden test. Compare logits and a short training step against a reference attention on padded, packed, and variable-length batches before celebrating IO savings.

IO-aware kernels are the wrong lever if you are bound on network all-reduce or on a CPU runtime. They will not fix a 100k-token prompt you should not send. Do not vendor-kernel hop weekly on the production trainer without numerical checks. If you call a hosted API, you do not pick the kernel. Use fused attention when you own the GPU path, have tests, and context length is actually the memory bottleneck.