CODA: Optimizing Transformer Blocks through GEMM-Epilogue Fusion

Modern Transformer training systems are built upon the foundation of dense linear algebra. However, as these systems scale, a significant portion of end-to-end execution time is no longer spent on the core matrix multiplications themselves, but on the surrounding "memory-bound" operators. Normalization, activations, residual updates, and reductions frequently move massive intermediate tensors through global memory while performing very little arithmetic, creating a critical bottleneck in otherwise highly optimized training stacks.

To address this, researchers have introduced CODA, a GPU kernel abstraction designed to rewrite Transformer blocks as "GEMM-plus-epilogue" programs. By reparameterizing how these operators are executed, CODA minimizes expensive data movement and maximizes hardware efficiency.

The Memory Wall in Transformer Blocks

In a standard Transformer forward or backward pass, the sequence of operations is typically: GEMM $\rightarrow$ Activation/Normalization $\rightarrow$ Residual Addition $\rightarrow$ GEMM.

In traditional framework-level execution (such as standard PyTorch or TensorFlow), each of these steps is often launched as a separate kernel. This means the output of a GEMM is written to global memory, read back by the normalization kernel, written again, and then read by the residual update kernel. This "round-trip" to global memory is the costliest part of the operation, as the arithmetic intensity of normalization and activations is extremely low compared to the bandwidth cost of moving the data.

How CODA Works: The Epilogue Abstraction

CODA's core insight is that many of these separate framework kernels can be algebraically reparameterized to execute while the GEMM output tile is still residing on the chip (in shared memory or registers), before it is ever written to global memory.

Instead of treating the GEMM as a black box, CODA fixes the GEMM mainloop and exposes a set of composable epilogue primitives. These primitives allow developers to define scaling, reductions, pairwise transformations, and accumulations directly within the GEMM's final stage.

Key Technical Insights

One of the most significant optimizations enabled by CODA involves the handling of row-wise dependent functions like RMSNorm or LayerNorm. As noted by technical observers, these functions often have baked-in scales that are commutative in specific setups.

((W1 @ gamma * globally_computed_scale) * W2 can be written as (W1 @ gamma * W2) * globally_computed_scale as long as we have row-only interactions for the scale.

This allows the scale to be moved after a subsequent projection, partially aggregated on tiles of rows, and fused into the epilogue. This is a powerful shift because standard left-to-right graph compilers (like torch.compile) often cannot assume that a global row-wise reduction between GEMMs is commutative, preventing them from performing this optimization automatically.

Productivity vs. Efficiency: The Role of LLMs

Writing high-performance GPU kernels manually—using CUDA or CUTLASS—is notoriously difficult and error-prone. CODA aims to bridge the gap between framework-level productivity and hardware-level efficiency by providing a constrained, composable API.

Interestingly, the research indicates that both human-authored and LLM-authored CODA kernels achieve high performance. This suggests a strategic shift in how we develop low-level kernels. While LLMs may still struggle with the intricate details of low-level hardware optimization (like register pressure or warp scheduling), they are excellent at high-level composition.

By providing a restricted API of expert-written blocks, CODA allows an LLM to "glue" these blocks together to create a complex kernel without needing to understand the underlying hardware intricacies. This effectively turns kernel development into a composition problem rather than a manual optimization problem.

Conclusion

CODA represents a practical path forward for the next generation of AI accelerators. By treating Transformer blocks as GEMM-epilogue programs, it eliminates the memory-bound bottlenecks that plague large-scale training. Furthermore, by simplifying the interface for kernel generation, it opens the door for agentic development where LLMs can autonomously optimize the computation graph of a model to maximize throughput on specific hardware.