Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP

Hugging Face has detailed the process of optimizing a Multilayer Perceptron (MLP) in PyTorch, demonstrating how to move from standard nn.Linear layers to fused kernels to reduce CPU overhead and GPU memory traffic. The primary takeaway is that while torch.compile can fuse pointwise operations into a single Triton kernel to avoid expensive HBM round-trips, hand-tuned kernels can provide similar performance gains without the compilation latency and shape-specialization overhead of the compiler.

The Mechanics of nn.Linear and Kernel Epilogues

In PyTorch, nn.Linear is a wrapper around matrix multiplication and addition. When bias=True, the operation is executed as y = x @ w.T + b.

Bias Folding via Epilogues

Profiling reveals that there is no separate aten::add kernel for the bias addition. Instead, the bias addition is folded into the matrix multiplication kernel using an epilogue. An epilogue is a small computation performed by a GEMM (General Matrix Multiply) kernel just before writing results back to High Bandwidth Memory (HBM). By integrating the bias add into the matmul kernel's writeback, PyTorch avoids loading and writing to HBM a second time, reducing memory traffic.

CPU Dispatch and Transpose Views

An aten::t (transpose) operation appears in the eager CPU dispatch chain before aten::addmm. However, this does not launch a GPU kernel. aten::t merely rewrites tensor metadata (shape and stride) on the CPU to create a new view of the same raw data.

When torch.compile is applied to a single nn.Linear layer, it does not change the GPU kernel used—the same cuBLAS GEMM kernel is executed. Instead, it removes the CPU overhead of dispatching the transpose view by hard-coding the resulting strides at compile time.

Profiling the GeGLU MLP

A GeGLU MLP consists of three linear projections (gate_proj, up_proj, and down_proj) and a pointwise activation sequence (GeLU followed by a multiplication).

Eager Execution Performance

In eager mode, a GeGLU MLP forward pass launches five distinct GPU kernels: three GEMMs and two pointwise kernels (GeLU and multiplication). The intermediate tensor produced by the GeLU operation must be written to HBM and then read back by the multiplication kernel, creating a memory bottleneck.

GEMM Kernel Variation

Not all GEMM kernels are identical even if they have the same FLOP count. For example, down_proj may be faster than gate_proj because cuBLAS selects a different tile size (e.g., 128x256 vs 128x128) and pipeline depth based on the input shapes to optimize data reuse.

Optimization via torch.compile and Fused Kernels

The Impact of torch.compile

torch.compile optimizes the MLP by collapsing the GeLU, multiplication, and reshape operations into a single fused Triton kernel (e.g., triton_poi_fused__unsafe_view_gelu_mul_0).

This fusion provides a significant performance win by keeping intermediate values in registers rather than writing them to HBM. However, this optimization comes with costs: the compiler introduces "pre-ops" such as TorchDynamo guards and prologue overhead, and it specializes the kernel for a specific input shape. If the input shape changes, the model must undergo re-tracing and recompilation.

Hand-Tuned Kernels with the kernels Library

To avoid compilation latency and shape-specialization, hand-tuned kernels can be used via the Hugging Face kernels library. For example, the LigerGEGLUMLP layer uses a pre-built Triton kernel that bakes in the fusion of GeLU and multiplication.

Comparison of Compiled vs. Hand-Tuned Kernels:

Feature	torch.compile (Inductor)	Hand-Tuned (Liger)
Fusion	Dynamic fusion of pointwise ops	Baked-in fusion
CPU Overhead	Dynamo guards and prologue	No compile pre-ops
Shape Handling	Specialized for static shapes (fastest for one shape)	Generic launch parameters (robust to shape changes)
Deployment	Requires local compilation/Triton	Pre-built binaries via HF Hub

While a compiled kernel might be slightly faster for a specific static shape due to extreme specialization, the hand-tuned kernel is more robust and avoids the risks and latency associated with the PyTorch compilation pipeline.

Summary of Optimization Stages

Setup	GPU Change	CPU Change
Eager nn.Linear	Bias add folded into GEMM epilogue	Baseline dispatch
Compiled nn.Linear	No change (same cuBLAS kernel)	Removed aten::t view bookkeeping
Eager MLP	5 kernels; intermediate results hit HBM	Baseline dispatch
Compiled MLP	GeLU + mul fused into one Triton kernel	Added Dynamo/guard pre-ops
Liger MLP	Same fusion as compiled; tuned launch params	No compile pre-ops; no recompilation risk