Ollama OpenAI Compatibility Update

Ollama now features built-in compatibility with the OpenAI Chat Completions API. This update enables developers to use a wide array of existing tooling and applications designed for OpenAI by simply redirecting them to a local Ollama instance.

Local Integration via OpenAI API Format

Ollama allows users to invoke local models using the standard OpenAI API format by changing the hostname to http://localhost:11434. This compatibility ensures that applications expecting an OpenAI-style endpoint can communicate with models like Llama 2 or Mistral running locally.

Implementation via cURL

To interact with the API using cURL, requests are sent to the /v1/chat/completions endpoint. A typical request includes the model name and a messages array containing system and user roles:

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama2",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

Implementation via Official Libraries

Ollama is compatible with the official OpenAI Python and JavaScript libraries. Integration requires setting the base_url (or baseURL) to http://localhost:11434/v1. While an api_key is required by the libraries to initialize the client, it is unused by Ollama.

Python Example:

from openai import OpenAI

client = OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama',
)

response = client.chat.completions.create(
  model="llama2",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The LA Dodgers won in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)
print(response.choices[0].message.content)

JavaScript Example:

import OpenAI from 'openai'

const openai = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',
})

const completion = await openai.chat.completions.create({
  model: 'llama2',
  messages: [{ role: 'user', content: 'Why is the sky blue?' }],
})

console.log(completion.choices[0].message.content)

Ecosystem Compatibility and Frameworks

Because Ollama adopts the OpenAI API standard, it integrates directly with popular open-source AI frameworks.

Vercel AI SDK

Developers building conversational streaming applications can use the Vercel AI SDK by updating the OpenAI client configuration in their API routes to point to the local Ollama base URL and specifying the desired local model (e.g., llama2) with stream: true.

Microsoft AutoGen

Ollama supports multi-agent application development via Microsoft AutoGen. By configuring the config_list with the local base URL and a model such as codellama, AutoGen agents can execute tasks locally, such as writing code to plot stock price changes.

Future Development Roadmap

The current OpenAI API support is initial and experimental. Ollama is considering the following improvements for future releases:

• Embeddings API: Support for generating vector embeddings.
• Function Calling: Ability for models to trigger external functions.
• Vision Support: Integration of multimodal capabilities.
• Logprobs: Provision of log probabilities for token generation.