Dao-AILab/causal-conv1d

Causal depthwise conv1d in CUDA, with a PyTorch interface

What it solves

It provides a highly optimized CUDA implementation of causal depthwise 1D convolutions, which are essential for sequence modeling in AI models. It replaces the standard PyTorch conv1d operation with a specialized kernel that is more efficient for causal operations (where the output at a time step depends only on the past and the past current step).

How it works

The project implements a custom CUDA kernel that performs depthwise 1D convolutions. It supports multiple precision formats (fp32, fp16, bf16) and kernel sizes of 2, 3, and 4. It integrates directly into PyTorch as a function causal_conv1d_fn that takes an input tensor, weights, and an optional bias and activation function (such as SiLU or Swish).

Who it’s for

Developers and researchers building high-performance sequence models or state-space models (SSMs) who need an efficient, low-level CUDA implementation of causal convolutions.

Highlights

  • Optimized CUDA kernels for causal depthwise 1D convolutions.
  • Support for fp32, fp16, and bf16 precision.
  • Support for kernel sizes 2, 3, and 4.
  • Built-in support for optional SiLU and Swish activation functions.
  • Compatibility with both NVIDIA (CUDA) and AMD (ROCm) hardware.

Related

  • Project
    MooreThreads/torch_musa

    torch_musa is a PyTorch extension that adds a **MUSA** backend for Moore Threads GPUs. It mirrors the standard PyTorch API, so you only replace `cuda` with `musa`. The package includes tools for building C/C++ extensions, distributed training via the **mccl** backend, and Docker images for easy setup. It also provides MUSA‑compatible builds of torchvision, torchaudio, and a curated list of other popular PyTorch libraries. Installation can be done from pre‑built wheels or by building from source after installing the MUSA SDK. The project is BSD‑licensed.

  • Project
    xlite-dev/LeetCUDA

    LeetCUDA is a GPL‑v3 repository that provides a 400‑page learning book and over 200 CUDA kernels (including high‑performance HGEMM and Flash‑Attention implementations) wrapped for PyTorch, targeting AI workloads on NVIDIA Tensor‑Core GPUs.

  • Project
    facebookresearch/pytorch3d

    A PyTorch-based library for 3D Computer Vision research providing differentiable rendering and efficient tools for manipulating 3D meshes and point clouds.

  • Project
    microsoft/microxcaling

    A PyTorch emulation library for MX-compatible formats and bfloat quantization, enabling data scientists to explore the impact of low-precision numerical formats on DNNs.

  • Project
    NVIDIA/cuEquivariance

    An NVIDIA Python library for building high-performance geometric neural networks that respect 3D symmetries using optimized CUDA kernels and segmented polynomials.