deepseek-ai/DeepSelect

What it solves

DeepSelect is a high-performance implementation of the TopK kernel, designed to replace torch.topk in specific AI workloads. It addresses the performance bottlenecks of the standard PyTorch implementation, providing a 2x to 20x speedup in memory bandwidth efficiency for selecting the top-k elements from a tensor.

How it works

DeepSelect provides optimized CUDA kernels for two primary scenarios: a "Lightning Indexer" for bfloat16 inputs (used in DeepSeek Sparse Attention) and a "Sampling Scenario" for float32 inputs (used in LLM sampling). It optimizes memory bandwidth and allows users to disable sorting or skip value output to further increase performance. It also supports variable-length rows and built-in NaN checking.

Who it’s for

It is intended for developers and researchers working with DeepSeek models (such as V3.2, V4, and V4.1) or anyone needing highly optimized TopK operations for large-scale LLM inference and training.

Highlights

• Significant Speedup: Achieves 2x to 20x faster performance compared to torch.topk.
• Specialized Scenarios: Optimized for bfloat16 (Sparse Attention) and float32 (Sampling) dtypes.
• Flexible Output: Option to skip value output or disable sorted indices to maximize speed.
• Variable Length Support: Handles rows of different lengths using an end tensor.
• Robustness: Integrated NaN checking to prevent silent failures during computation.