kkokosa/dotLLM

LLM inference engine written in .NET

What it solves

dotLLM is a high-performance LLM inference engine built natively in C# and .NET. It eliminates the need for Python wrappers or external libraries like llama.cpp, providing a pure .NET implementation for running transformer-based models (such as Llama, Mistral, Phi, Qwen, and DeepSeek) with native speed and efficiency.

How it works

The engine uses unmanaged memory to achieve zero-GC inference and leverages SIMD vectorization for CPU compute. For GPU acceleration, it uses CUDA PTX kernels loaded via the CUDA Driver API. It supports GGUF model loading through memory-mapped files for near-instant startup. The architecture is layered, separating the server (OpenAI-compatible API), the engine (KV-cache and scheduling), and the backends (CPU/CUDA).

Who it’s for

.NET developers who want to integrate LLM inference directly into their applications without relying on Python or native shared libraries, as well as users looking for a lightweight, high-performance local LLM server with an OpenAI-compatible API.

Highlights

  • Native .NET Implementation: Written from the ground up in C#; not a wrapper.
  • Performance Optimizations: Zero-GC hot path, SIMD-optimized CPU kernels, and CUDA GPU acceleration.
  • Advanced Inference Features: Supports speculative decoding, paged KV-cache, and prompt caching.
  • Structured Output: Constrained decoding for guaranteed JSON, JSON Schema, regex, and grammar compliance.
  • Deployment Flexibility: Available as a global .NET tool, self-contained binaries, or NuGet packages.
  • OpenAI Compatibility: Includes an ASP.NET server with /v1/chat/completions and a built-in web UI.

Related

  • Project
    SciSharp/LLamaSharp

    A cross-platform C# library based on llama.cpp that allows developers to run LLaMA and other LLMs locally on CPU and GPU with high-level APIs.

  • Project
    ggml-org/llama.cpp

    A plain C/C++ implementation for high-performance LLM and VLM inference across a wide range of hardware with minimal setup.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.

  • Project
    zhongkaifu/TensorSharp

    TensorSharp is a native .NET engine that runs GGUF‑format LLMs, diffusion‑based image/video generation, and multimodal models on CUDA, Vulkan, Metal, Apple‑MLX, or pure C# CPU. It matches or beats llama.cpp, supports continuous batching, paged KV caches, tensor‑parallelism, speculative decoding, tool‑calling/agent skills, and provides Ollama/OpenAI‑compatible APIs and a web UI.