kkokosa/dotLLM

LLM inference engine written in .NET

What it solves

dotLLM is a high-performance LLM inference engine built natively in C#/.NET. It eliminates the need for Python wrappers or dependencies on external libraries like llama.cpp, allowing .NET developers to run transformer-based models (such as Llama, Mistral, Phi, Qwen, and DeepSeek) directly within the .NET ecosystem with high efficiency.

How it works

The engine uses a layered architecture that separates the core tensor abstractions from the specific compute backends. It implements all orchestration, model loading, and tokenization in pure C#.

  • Compute Backends: It supports SIMD-optimized CPU inference and CUDA GPU acceleration via PTX kernels loaded through the CUDA Driver API.
  • Memory Management: To achieve high performance, it uses unmanaged memory for tensor data to avoid Garbage Collection (GC) overhead on the hot path and employs memory-mapped files for near-instant model loading of GGUF files.
  • Inference Optimizations: It features paged KV-cache (PagedAttention), speculative decoding for faster generation, and constrained decoding using FSM/PDA to guarantee structured outputs like JSON or regex.
  • Serving: It provides an OpenAI-compatible API server built with ASP.NET, including a built-in chat UI.

Who it’s for

It is designed for .NET developers who want to integrate LLM inference into their applications without relying on Python or native shared libraries, as well as users looking for a lightweight, high-performance local LLM runner for Windows, Linux, or macOS.

Highlights

  • Pure .NET Implementation: Not a wrapper; native C# implementation of the inference pipeline.
  • Zero-GC Inference: Uses unmanaged memory to prevent performance spikes from managed heap allocations.
  • Advanced Decoding: Supports speculative decoding and constrained decoding for guaranteed JSON/Schema/Regex outputs.
  • Hardware Acceleration: SIMD-optimized CPU kernels and CUDA GPU support with hybrid offloading.
  • GGUF Support: Native loading of GGUF quantized formats (FP16, Q8_0, Q4_K_M).
  • OpenAI Compatible: Includes an API server and CLI for easy integration and testing.