zhongkaifu/TensorSharp

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability

TensorSharp – A .NET‑based Local LLM Inference Engine

What it is – TensorSharp is a .NET 10 library that runs GGUF‑format LLMs locally. It can be used from a command‑line tool, a web‑server that mimics Ollama/OpenAI APIs, or embedded in any .NET application. The engine ships with C#‑only CPU kernels and native back‑ends for CUDA, Metal and Vulkan, letting you run on Windows, macOS and Linux – with or without a GPU.

Key capabilities

  • Model support – dozens of text, reasoning and multimodal families (Gemma 4, Qwen 3.x, DeepSeek V4, Mistral 3, etc.), plus diffusion‑style text‑to‑image, image editing, video‑with‑audio, and embedding models.
  • Back‑ends – ggml_cpu (pure C#), ggml_cpu native, ggml_cuda, ggml_metal, ggml_vulkan. You pick the one that matches your hardware.
  • Serving features – continuous batching, a paged KV cache with prefix sharing, speculative decoding, tensor‑parallelism, and tool‑calling/agent skills.
  • APIs – CLI (TensorSharp.Cli), a web UI/server (TensorSharp.Server.Host) exposing Ollama‑compatible and OpenAI‑compatible endpoints, and the TensorAgent desktop/mobile app that adds file‑, shell‑ and document‑tool workflows.
  • Performance – benchmarked against llama.cpp; on an RTX 3080 laptop GPU TensorSharp is 1.0‑1.3× faster for pre‑fill and decode on several models.

How to get started

  1. Download a pre‑built release for your OS (CPU or CUDA version).
  2. Grab a GGUF model, e.g. Gemma 4 E4B from HuggingFace.
  3. Run the CLI:
    dotnet run --project TensorSharp.Cli -- --model models\gemma-4-E4B-it-Q8_0.gguf \
    --backend ggml_cuda --input prompt.txt --max-tokens 128
    (use ggml_metal on Apple Silicon, ggml_vulkan for AMD/Intel GPUs, or ggml_cpu for CPU‑only).
  4. To expose a server, run TensorSharp.Server.Host – it listens on 0.0.0.0:5000 and offers a browser chat UI plus API endpoints.
  5. For a full desktop experience, install TensorAgent (DMG/PKG for macOS, MSI/ZIP for Windows). The app bundles the runtime, lets you download models, attach files, and run agent skills.

Who it’s for – .NET developers who want to run LLMs locally without relying on Python‑centric stacks, and anyone needing a single engine that works across CLI, web‑API and native desktop/mobile clients.

Documentation & resources – The repo includes extensive docs: quick‑start guide, backend selection table, model‑family cards, feature deep‑dives (continuous batching, speculative decoding, agent skills), benchmark tables, and a “book guide” that links to two Amazon‑published books covering the code in detail.

License – See the LICENSE file in the repository (standard open‑source terms).

All information above is taken directly from the repository’s README; no external assumptions have been added.

Related

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    NVIDIA/TensorRT-Edge-LLM

    TensorRT Edge‑LLM is NVIDIA’s C++/Python inference runtime for running LLMs and multimodal models on Jetson, DRIVE and DGX Spark edge hardware. It converts Hugging Face checkpoints to TensorRT engines (via ONNX or a direct builder), offers quantization, in‑flight batching, speculative decoding, and a vLLM‑style Python API with an OpenAI‑compatible server. The library targets automotive, robotics, industrial IoT and consumer edge use‑cases, delivering high‑throughput, low‑latency inference on resource‑constrained GPUs.

  • Project
    NVIDIA/TensorRT

    A high-performance inference optimizer and runtime from NVIDIA that accelerates AI model deployment and throughput on NVIDIA GPUs.

  • Project
    antirez/ds4

    A native inference engine optimized for running high-end LLMs like DeepSeek V4 and GLM 5.x on consumer hardware using SSD streaming and distributed inference.

  • Project
    kigner/audio.cpp-webui

    A high-performance C++ audio inference framework based on ggml that enables fast, portable local execution of TTS, ASR, and voice cloning models across multiple hardware backends.