Ultrafast Machine Learning on FPGAs via Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) can be implemented on Field-Programmable Gate Arrays (FPGAs) using lookup tables (LUTs) to achieve sub-microsecond inference and real-time on-chip learning. This approach leverages the univariate nature of KAN activations and the inherent hardware primitives of FPGAs to bypass the latency overheads associated with GPUs and CPUs.

KANs as Trainable Lookup-Table Neural Networks

KANs are uniquely suited for FPGA implementation because they replace the fixed activation functions and scalar weights of Multi-Layer Perceptrons (MLPs) with learnable univariate functions on the edges. This allows the network to be represented as a series of lookup tables (LUTs) rather than complex matrix multiplications.

Hardware Implementation of Inference

To perform inference on an FPGA, trained KAN activations are converted into LUTs. Because KANs sum univariate activations, they avoid the exponential resource scaling that typically plagues multivariate LUT-based networks. The architecture consists of LUTs for each activation, followed by an adder tree to compute the final sum. This design enables nanosecond-latency inference and a 2700x speedup over previous KAN-FPGA implementations.

Fixed-Point Quantization

FPGAs operate on bits, requiring real-valued functions to be quantized. The researchers used fixed-point quantization, where numbers are represented in base-2 with a fixed number of fractional bits. This process converts continuous functions into binary functions that can be stored directly in LUTs, minimizing quantization error to maintain model stability.

Real-Time On-FPGA Online Learning

Beyond static inference, KANs enable real-time gradient-based learning directly on the FPGA. This allows models to adapt to non-stationary systems—such as quantum control or nuclear fusion—where parameters must be updated within a fraction of a microsecond.

B-Spline Locality and Sparse Updates

To enable online learning, the system stores B-spline basis functions in LUTs rather than the final activations. B-splines are local basis functions, meaning only a small subset of them are nonzero for any given input.

This locality ensures that the hardware logic for forward and backward passes scales with the order of the spline (k) rather than the total number of grid cells (G). This allows the network to scale horizontally in expressivity without a proportional increase in hardware resource usage.

Stability in Fixed-Point Training

One of the primary challenges of FPGA training is the wide variance in weight and gradient magnitudes. KANs mitigate this through the boundedness of B-splines. Since the output of a KAN activation is always bounded between the smallest and largest coefficient, both activations and gradients remain within predictable ranges. This stability makes it easier to select an optimal quantization range, reducing error and improving learning stability compared to MLPs.

Performance and Use Cases

The KAN-based online learner can scale to over 50,000 parameters while maintaining sub-microsecond latencies for both forward and backward passes. This performance is particularly effective for function approximation, qubit readout, and non-stationary control.

Community Insights and Constraints

While the KAN-FPGA approach offers extreme latency benefits, community discussion highlights several constraints:

  • Throughput vs. Latency: The architecture is optimized for ultra-low latency rather than high throughput. It is not currently suitable for accelerating Large Language Models (LLMs), which require massive throughput and are too large for current FPGA capacities.
  • Model Size: The approach is limited to relatively small models or requires extremely large FPGAs to accommodate the LUT-based representation.
  • Hardware Synthesis: Some practitioners noted that FPGA synthesis is complex, as LUTs are not always simple 2:1 mappings and may include flip-flops and muxes, which can complicate the direct translation of activation functions.

"The time it would take for data to be sent to and from a CPU/GPU, let alone perform computation, already exceeds a microsecond in most cases."

This underscores why dedicated on-chip logic is necessary for applications requiring nanosecond-scale adaptation.

Sources