llama.cpp Model Management and Router Mode

The llama-server now includes a router mode that allows users to dynamically load, unload, and switch between multiple models without needing to restart the server. This update brings Ollama-style model management to llama.cpp, utilizing a multi-process architecture where each model runs in its own process to ensure that a crash in one model does not affect others.

Dynamic Model Loading and Routing

llama-server can now manage multiple models on-demand, routing requests to the appropriate model based on the model field in the API request.

Key Management Features

  • Auto-discovery: The server automatically scans for GGUF files in the llama.cpp cache (defined by LLAMA_CACHE or ~/.cache/llama.cpp) or a custom directory specified via --models-dir.
  • On-demand Loading: Models are loaded into memory automatically upon the first request. Subsequent requests to the same model are processed instantly.
  • LRU Eviction: To manage memory, the server employs a Least-Recently-Used (LRU) eviction policy. When the maximum number of loaded models (defined by --models-max, default is 4) is reached, the least recently used model is unloaded to free resources.
  • Manual Control: Users can explicitly manage models using the /models/load and /models/unload endpoints via HTTP POST requests.

Configuration and Technical Options

Models in router mode inherit global settings from the router, such as context size (-c) and GPU offloading (-ngl). However, granular control is available through presets.

Key Command-Line Flags

Flag Description
--models-dir PATH Specifies the directory containing GGUF files.
--models-max N Sets the maximum number of models loaded simultaneously (default: 4).
--no-models-autoload Disables automatic loading, requiring explicit /models/load calls.

Per-Model Presets

Users can define specific configurations for individual models using a config.ini file passed via --models-preset. This allows for custom context sizes and temperature settings per model. According to community discussion, presets.ini can also be used to specify mmproj (multimodal projector) files for specific models.

API and Interface Integration

Model management is integrated into both the API and the user interface:

  • API Listing: A GET request to /models returns all discovered models and their current status (loaded, loading, or unloaded).
  • Web UI: The built-in llama.cpp web UI supports model switching via a dropdown menu, which triggers automatic loading.
  • OpenAI Compatibility: The server remains a lightweight, OpenAI-compatible HTTP server, making it easy to integrate into existing workflows.

Community Insights

Users have highlighted the utility of this feature for A/B testing model versions and creating multi-tenant deployments. Regarding technical specifics, community members noted:

"llama-server by default in most implementation keeps the reasoning content in reasoning_content variable in response attribute. You can get it from there. Otherwise use reasoning-format flag and pass DeepSeek value to get pure tokens"

This suggests that while the router manages the models, the underlying server continues to support advanced output formats like reasoning tokens for specific model architectures.

Sources