GPUs were already massively parallel when they only drew triangles. The hardware hid that parallelism behind graphics APIs. CUDA, launched in 2006 and described for a systems audience in papers such as Nickolls et al., exposed a programming model: kernels launched over a grid of threads, grouped into warps that execute in lockstep (SIMT), with a memory hierarchy the programmer must respect — registers, shared memory, device DRAM, and later unified virtual memory. That model is why a generation of HPC codes, and then every major neural-net framework, compiles to NVIDIA GPUs instead of treating them as opaque shaders.

SIMT is not multithreaded CPU programming

A warp of 32 threads that take different branches serializes those paths (divergence). A kernel that looks like a CPU if tree can run at a fraction of peak. Coalesced memory access — neighboring threads touching neighboring addresses — is the other commandment. Random gathers from DRAM waste the bus. CUDA's career-making trick was making these rules teachable: occupancy calculators, nvprof/ncu, and a C dialect that looks familiar until you ignore the memory model.

Shared memory (the programmable L1-like scratchpad) exists so a tile of a matrix multiply can be reused by a block. cuBLAS and later Tensor Cores are what you should call instead of writing that tile yourself, but the conceptual unit — block-level cooperation, grid-level parallelism — is still how you debug why a custom kernel is slow. Host-device copies are the third footgun. Naive cudaMemcpy of every mini-batch kills training; streams, pinned memory, and (on newer architectures) better interconnects exist to overlap copy and compute.

Software stacks on top of the ISA

CUDA is a moving platform: compute capabilities, deprecated APIs, and a driver/runtime split that production clusters have to pin. Frameworks (PyTorch, TensorFlow) generate kernels or call libraries so most application engineers never write a kernel. They still inherit CUDA's failure modes: OOM without a clear culprit, nondeterministic reductions, and a single GPU fault that kills a process. MPS and MIG partition GPUs for multi-tenant serving; they do not make an oversized model fit.

Error handling is historically lazy in tutorials (cudaMalloc unchecked). At scale, a device-side assert or an illegal access surfaces asynchronously. cudaDeviceSynchronize in debug and proper stream error checks in production are operational, not pedantic. Forward compatibility — an old binary on a new driver — usually works; the reverse does not. Container images that bundle user-mode libraries still need a matching host driver, which is why datacenter Kubernetes GPU plugins exist.

The borrow for teams who will never write CUDA is still architectural: move data-parallel loops to the device in large tiles, minimize host round trips, and treat memory bandwidth as the scarce resource. CUDA made that pattern the default for the industry.

Determinism is a late surprise. Atomic adds and warp shuffles used in reductions do not guarantee bitwise-identical results across runs or GPU SKUs. Scientific and finance teams need a policy: bitwise, or tolerant of ULP error. Mixed precision (TF32, FP16) makes that policy sharper. Capture it in tests before a kernel "optimization" ships as a silent numeric change.

What you can borrow

  • Structure compute as regular, coalesced, low-divergence parallel loops; branchy code belongs on the CPU.
  • Reuse data in on-chip scratchpads or library kernels; do not refill DRAM per inner iteration.
  • Overlap copies with compute; synchronous memcpy of every batch is a hidden sequential bottleneck.
  • Pin driver, toolkit, and framework versions in clusters; GPU stacks are not "just pip install."
  • Check asynchronous device errors. GPU faults do not always raise on the next host line.