What Happens When You Run a CUDA Kernel: From Source to SASS

Running a CUDA kernel involves a complex transition from high-level C++ code to machine-level instructions executed across thousands of GPU cores. The process spans multiple compilation stages, a specialized driver communication protocol involving "doorbells" and "pushbuffers," and a hardware scheduling system that hides memory latency through massive parallelism.

The Compilation Pipeline: From Source to SASS

CUDA code is not compiled directly into a single binary but passes through a multi-stage pipeline managed by the nvcc driver program. This process ensures both performance for specific architectures and forward compatibility.

PTX: The Virtual ISA

nvcc first uses cicc (an LLVM-based compiler) to translate the CUDA source into PTX (Parallel Thread Execution). PTX is a virtual Instruction Set Architecture (ISA) that is device-agnostic and uses an infinite number of typed registers. Because it is agnostic, PTX is often verbose; for example, calculating a memory address may take several instructions to ensure the pointer is converted to a global address and the index is widened to 64 bits.

SASS: The Hardware Machine Code

The ptxas assembler then converts PTX into SASS (Streaming Assembler), which is the actual machine code for a specific GPU architecture (e.g., sm_89 for the RTX 4090). During this transition, the compiler optimizes the code by:

Register Allocation: Collapsing virtual PTX registers into a limited set of physical registers.
Instruction Fusion: Merging multiple PTX operations (like multiply and add) into a single IMAD.WIDE SASS instruction.
Address Simplification: Absorbing pointer conversions directly into the addressing modes of the instructions.

The Fatbinary

To maintain compatibility, nvcc bundles both the SASS (in a .cubin ELF file) and the compressed PTX into a fatbinary. This fatbinary is embedded into the host executable. If the program is run on a GPU architecture for which SASS was not pre-compiled, the NVIDIA driver can JIT-compile the embedded PTX into the appropriate SASS at load time.

Triggering the GPU: The Host-to-Device Bridge

Launching a kernel requires bridging the host CPU and the GPU across the PCIe bus. This is handled by a combination of the CUDA runtime and the NVIDIA user-mode driver (libcuda.so).

Host Launch Stubs

When the compiler encounters a kernel launch (e.g., vadd<<<grid, block>>>), it replaces it with a host launch stub. This stub packs the kernel arguments into a buffer in host memory at specific byte offsets. These offsets correspond to the constant bank offsets that the SASS code will later read from on the GPU.

The Driver Conversation

The CUDA runtime uses the host-side function pointer as a lookup key to find the device-side symbol in the fatbinary. It then calls the user-mode driver, which communicates with the kernel-mode driver (nvidia.ko) via ioctl calls on device files like /dev/nvidiactl.

The Execution Mechanism: Pushbuffers and Doorbells

GPUs do not accept function calls; they read a stream of commands from host memory. The driver manages this via a "channel" consisting of two primary structures in host RAM:

The Pushbuffer: A region where the driver writes "methods" (register addresses and values) that define GPU actions.
The GPFIFO: A ring buffer of pointers that tells the GPU which spans of the pushbuffer to read.

The Launch Sequence

To launch a kernel, the driver:

Writes the Queue Meta Data (QMD) into the pushbuffer. The QMD is the launch descriptor containing grid/block dimensions, register requirements, and pointers to the SASS code and the constant bank of arguments.
Updates the GP_PUT cursor to signal new work is available.
Rings the Doorbell: The driver writes a work-submit token to a memory-mapped register on the GPU. This alerts the GPU's host engine to fetch the methods from the pushbuffer via DMA.

Hardware Execution: SMs and Warp Scheduling

Once the host engine receives the QMD, it hands it to the Compute Work Distributor (GigaThread Engine), which distributes blocks of threads across the GPU's Streaming Multiprocessors (SMs).

Resource Constraints and Occupancy

The number of blocks an SM can host is limited by hardware caps. For an Ada Lovelace SM, the bottlenecks are typically:

Register Capacity: Total registers available divided by registers required per thread.
Thread Capacity: A hard limit on the maximum number of active threads per SM (e.g., 1,536 threads).

Warp Eligibility and Latency Hiding

GPUs hide memory latency by switching between many resident warps. A warp is "eligible" to run based on metadata packed into the SASS instructions by ptxas:

Static Stall Counts: For fixed-latency math, the compiler encodes exactly how many cycles the warp must wait before the next instruction.
Scoreboard Barriers: For variable-latency operations (like LDG global loads), the hardware uses six physical scoreboard barriers. An instruction "sets" a barrier when it starts a load and a subsequent instruction "waits" on that barrier. The warp remains ineligible until the barrier clears.
Yield Hints: A bit that tells the scheduler to prioritize other warps if a bottleneck is imminent.

Memory Hierarchy and Data Movement

For a simple vector addition, the GPU performs request coalescing, merging 32 individual 4-byte thread requests into four 32-byte sector requests to maximize bandwidth.

Data flows from VRAM $\rightarrow$ L2 Cache $\rightarrow$ L1 Cache $\rightarrow$ Registers. Because the arithmetic intensity of vector addition is extremely low, the performance is almost entirely bound by DRAM throughput. In many cases, the output data remains "dirty" in the L2 cache and is transferred back to the host via DMA without ever being written back to physical VRAM.

Summary of the Full Path

Stage	Action	Key Component
Compile	Source $\rightarrow$ PTX $\rightarrow$ SASS $\rightarrow$ Fatbin	`nvcc`, `ptxas`
Launch	Pack Args $\rightarrow$ QMD $\rightarrow$ Pushbuffer $\rightarrow$ Doorbell	`libcuda`, `nvidia.ko`
Schedule	Distribute Blocks $\rightarrow$ SM $\rightarrow$ Warp Scheduler	GigaThread Engine
Execute	Coalesced Load $\rightarrow$ FADD $\rightarrow$ Store	SM, L1/L2 Cache, VRAM
Return	Completion Semaphore $\rightarrow$ DMA Copy $\rightarrow$ Host	Copy Engine, PCIe

Community Insights

Discussion around this pipeline highlights that while the CUDA Runtime API simplifies this process through implicit synchronization, the CUDA Driver API provides more transparency for developers who wish to compile kernels at runtime and have better visibility into the underlying hardware interactions.