deepseek-ai/DeepJIT

What it solves

DeepJIT provides a lightweight, header-only C++20 runtime that simplifies the process of Just-In-Time (JIT) compilation for GPU and NPU kernels. It removes the need for kernel library authors to build their own infrastructure for compiling source code at runtime, caching binaries, and loading them onto hardware devices.

How it works

DeepJIT offers a shared interface for two primary backends: NVIDIA CUDA GPUs and HUAWEI Ascend NPUs. It manages the full lifecycle of a kernel: compiling source code using the appropriate toolchain (NVCC for CUDA, Bisheng/ld.lld for Ascend), caching the resulting binaries on disk and in memory to avoid redundant compilations, and launching the kernels with backend-specific options.

Key technical mechanisms include:

• Hashing and Caching: It generates unique cache keys based on the source code, tracked include files, compiler versions, and configuration options. This allows binaries to be reused across different processes, users, and even different nodes via shared filesystems.
• Include Parser: A line-oriented scanner tracks specific angle-bracket includes to ensure that changes in dependency headers trigger a re-compile.
• PyTorch Integration: It integrates with PyTorch's CUDA or torch_npu streams and can be exposed to Python via pybind11.
• Lazy Initialization: Device and compiler discovery are deferred until the runtime is actually used to reduce startup overhead.

Who it’s for

It is designed for C++ and Python extension authors who write high-performance device kernels and want to provide a JIT compilation experience for their users without implementing the low-level plumbing for compilation and caching.

Highlights

• Multi-Backend Support: Unified workflow for both NVIDIA CUDA and HUAWEI Ascend NPUs.
• Distributed Caching: Supports sharing compiled kernels across multiple users and nodes using POSIX-compliant filesystems.
• PyTorch Ready: Seamlessly integrates with PyTorch streams and provides a get_jit() Python API.
• Compilation Diagnostics: Ability to dump PTX/SASS (CUDA) or assembly (Ascend) for debugging and inspection.
• Post-Compilation Hooks: CUDA backend supports running custom Python scripts to modify the CUBIN file before it is cached.