Ollama Official Docker Image Release

Ollama is now available as an official Docker sponsored open-source image, allowing users to run large language models (LLMs) locally within Docker containers to ensure private data remains on the local machine without being sent to third-party services.

GPU Acceleration and Platform Support

Ollama provides a CLI and a REST API for application interaction, with hardware acceleration support varying by platform:

Linux

On Linux, Ollama supports GPU acceleration inside Docker containers specifically for Nvidia GPUs. To enable this, users must install the Nvidia container toolkit before running the container with the --gpus=all flag.

macOS

On macOS, users should run Ollama as a standalone application outside of Docker containers. This is because Docker Desktop does not currently support GPU acceleration, which is necessary for optimal model performance.

Deployment and Configuration

Ollama can be deployed via Docker using different configurations based on available hardware:

CPU-Only Deployment

To run Ollama using only the CPU, use the following command:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Nvidia GPU Deployment

After installing the Nvidia container toolkit, use the following command to enable GPU acceleration:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Running Models within the Container

Once the Ollama container is active, users can execute models such as Llama 2 directly within the container using the docker exec command:

docker exec -it ollama ollama run llama2

Additional models are available via the Ollama library.