zhongkaifu/TensorSharp

What it solves

TensorSharp is a native .NET inference engine designed to run large language models (LLMs) and multimodal models in the GGUF format. It provides a high-performance alternative to C++ based engines like llama.cpp, allowing .NET developers to deploy AI models with native performance on various hardware backends without relying on heavy external dependencies.

How it works

The engine implements a wide range of compute backends including GGML (Metal, CUDA, Vulkan), a direct CUDA/cuBLAS path, MLX for Apple Silicon, and a pure-C# CPU path. It utilizes advanced optimization techniques such as continuous batching, paged KV caching (vLLM-style), and speculative decoding to increase throughput and reduce latency. For very large models, it supports tensor parallelism and distributed clustering, enabling a single model to be split across multiple GPUs or multiple machines via TCP.

Who it’s for

It is primarily for .NET developers and AI engineers who want to run multimodal LLMs locally or as a server with OpenAI-compatible APIs, as well as those looking for a high-performance, native .NET implementation of model inference.

Highlights

• Multimodal Capabilities: Supports autoregressive LLMs, DiffusionGemma text-diffusion, Qwen-Image-Edit for image editing, and MiniMax-H3 for joint audio-video generation.
• High Performance: Matches or exceeds llama.cpp performance on specific workloads (e.g., Gemma 4 E4B) using pure .NET.
• Advanced Scaling: Features tensor parallelism (--tp) and peer-to-peer TCP clustering for distributed inference across nodes.
• Speculative Decoding: Includes four different algorithms to accelerate token generation, including MTP-retaining GGUFs and n-gram matching.
• Flexible Deployment: Ships with a console application, a browser-based chat UI, and Ollama/OpenAI-compatible HTTP APIs.
• Broad Hardware Support: Runs on NVIDIA (CUDA), Apple Silicon (Metal/MLX), and various GPUs via Vulkan, with a pure-C# CPU fallback.