SciSharp/LLamaSharp

What it solves

LLamaSharp provides a way to run Large Language Models (LLMs) like LLaMA locally on a device without needing to compile C++ code manually. It simplifies the deployment of LLMs into applications by providing high-level APIs and support for Retrieval-Augmented Generation (RAG).

How it works

It is a C# library based on llama.cpp, allowing for efficient inference on both CPUs and GPUs. It uses native "backends" (pre-compiled libraries for Windows, Linux, and Mac) to handle the heavy lifting of model execution. The library supports the GGUF model format, which is a quantized version of models originally in PyTorch or Huggingface formats to reduce memory usage.

Who it’s for

Developers who want to integrate local LLMs into their .NET applications, including those building desktop apps (WPF), web apps (ASP.NET, Blazor), or game engines (Unity).

Highlights

• Cross-platform support: Works on Windows, Linux, and Mac.
• Hardware acceleration: Supports CPU, CUDA, Metal, and Vulkan backends for GPU offloading.
• RAG support: Includes built-in capabilities for Retrieval-Augmented Generation.
• Multimodal capabilities: Supports multimodal models (e.g., LLaVA).
• Broad model compatibility: Compatible with a wide range of models including Llama 3.1, Gemma, and DeepSeek R1 via GGUF.