docker/model-runner

Docker Model Runner

What it solves

Docker Model Runner (DMR) simplifies the management, execution, and deployment of AI models. It removes the friction of manually configuring inference environments by allowing developers to pull and run large language models (LLMs) directly from Docker Hub or other OCI-compliant registries using a standardized CLI.

How it works

It operates as an inference daemon and CLI tool (integrated into Docker Desktop/Engine or available as a standalone dmr binary). It manages the lifecycle of models—pulling, listing, and running them—and provides a REST API for interaction. The system integrates multiple inference backends, including llama.cpp (with CPU, CUDA, and ROCm variants) and vLLM, to handle the actual model execution and GPU acceleration.

Who it’s for

Developers who want to run and serve AI models locally or in production without dealing with the complexities of manual environment setup, GPU driver configuration, or backend-specific boilerplate.

Highlights

  • Flexible Installation: Available as a Docker plugin, part of Docker Desktop/Engine, or a standalone binary that requires no Docker installation.
  • Multiple Backends: Supports llama.cpp and vLLM for inference.
  • Hardware Acceleration: Automatic GPU detection and support for NVIDIA (CUDA) and AMD (ROCm) GPUs.
  • Standardized API: Exposes a REST API for chatting with models and managing model lifecycles.
  • Orchestration: Includes dmrlet, a specialized orchestrator for stateless AI inference workloads that simplifies multi-GPU mapping.

Related

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    eugr/spark-vllm-docker

    A Docker-based deployment framework for vLLM optimized for NVIDIA DGX Spark, providing tools for single- and multi-node cluster setup, model distribution, and hardware-specific performance tuning.

  • Project
    n8n-io/self-hosted-ai-starter-kit

    A Docker Compose template that bundles n8n, Ollama, and Qdrant to quickly set up a self-hosted, local AI development environment for building AI agents and workflows.

  • Project
    eleiton/ollama-intel-arc

    A Docker-based deployment suite that enables Intel Arc GPU acceleration for Ollama, Stable Diffusion, and OpenAI Whisper via a unified interface.

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.