rafska/awesome-local-llm

A curated list of awesome platforms, tools, practices and resources that helps run LLMs locally

What it solves

Running Large Language Models (LLMs) on your own hardware can be overwhelming due to the sheer number of available tools, engines, and models. This project provides a centralized, curated directory to help users navigate the ecosystem of local AI, from choosing an inference engine to finding the right model for specific tasks like coding or multimodal interaction.

How it works

The project serves as a comprehensive "awesome" list—a structured repository of links and brief descriptions. It categorizes resources into logical groups such as inference platforms (for running models), inference engines (the underlying technology that executes models), user interfaces, specific model collections, and hardware recommendations. It also includes sections for tutorials and community resources to support the learning process.

Who it’s for

This list is designed for developers, researchers, and AI enthusiasts who want to run LLMs locally for privacy, cost-efficiency, or customization. It is equally useful for beginners looking for an easy "all-in-one" application and power users seeking high-performance serving engines or specialized models for agentic workflows.

Highlights

  • Inference Platforms: A curated selection of user-friendly tools like LM Studio, Ollama, and Jan for running models locally.
  • Inference Engines: A deep dive into the technical backends that power local AI, including llama.cpp, vLLM, and hardware-specific optimizations for Apple Silicon, NVIDIA, and AMD.
  • Model Directory: Categorized lists of models for general purpose, coding, vision, and audio tasks, including collections from major labs like Mistral, NVIDIA, and Qwen.
  • Agent & Tooling Ecosystem: Resources for building AI agents, including frameworks for browser automation, memory management, and RAG (Retrieval-Augmented Generation).
  • Learning Resources: A collection of tutorials covering prompt engineering, context management, and model training/fine-tuning.

Related

  • Project
    antirez/h3.c

    A native Metal inference engine for MiniMax-H3 on Apple Silicon, enabling local text-to-video and text-to-audio generation with advanced memory and performance optimizations.

  • Project
    thu-pacman/chitu

    A production-grade high-performance inference engine that supports diverse hardware backends and scalable deployment for large language models.

  • Project
    pytorch/ao

    TorchAO is a PyTorch-native quantization library that lets you compress and speed up models with minimal code changes. It supports float8 training, int4/int8 inference, and quantization-aware training to recover accuracy, delivering up to 1.5-2x speedups across popular workloads.

  • Project
    pytorch/TensorRT

    Torch‑TensorRT is an open‑source library that lets you compile any PyTorch model to run with TensorRT speed on NVIDIA GPUs. Use a single line (`torch.compile(..., backend="tensorrt")`) or an export workflow to produce a serialized module that can be loaded in Python or C++. It supports Linux, Windows, and Jetson, works with CUDA 13.4 + TensorRT 11.3, and is installable via pip or NVIDIA containers. Ideal for developers who want fast inference for vision, diffusion, or LLM models without leaving the PyTorch ecosystem.