Ollama adds streaming tool calling support

TL;DR

Ollama introduced streaming tool calling, enabling chat apps to receive token‑by‑token model output while simultaneously invoking functions such as weather lookups or mathematical calculations.

What streaming tool calling enables

Streaming tool calling lets a model interleave regular text with function calls without waiting for the entire response to finish. Applications can display partial content to users instantly and execute tools (e.g., API calls) as soon as the model signals a call.

Models that support tool calling

The following Ollama‑hosted models are compatible with the new streaming feature:

Qwen 3
Devstral
Qwen2.5 and Qwen2.5‑coder
Llama 3.1
Llama 4
Additional models listed under the "tool calling" filter on the Ollama model library.

Simple tool‑calling example (weather)

A cURL request streams a weather query using the get_current_weather function. The response includes incremental JSON fragments and a tool_calls field as soon as the model decides to invoke the function.

{
  "model": "qwen3",
  "created_at": "2025-05-27T22:54:58.100509Z",
  "message": {
    "role": "assistant",
    "content": "",
    "tool_calls": [{
      "function": {
        "name": "get_current_weather",
        "arguments": {"format": "celsius", "location": "Toronto"}
      }
    }]
  },
  "done": false
}

The stream continues with the function result once it is available.

Python SDK usage

The Python library (pip install -U ollama) accepts native Python callables as tools. In the example below, a simple add_two_numbers function is passed to chat() with stream=True. The client prints streamed text and detects tool_calls as they appear.

from ollama import chat, ChatResponse

def add_two_numbers(a: int, b: int) -> int:
    return a + b

messages = [{"role": "user", "content": "what is three minus one?"}]
response: ChatResponse = chat(model='qwen3', messages=messages, tools=[add_two_numbers], stream=True)

for chunk in response:
    print(chunk.message.content, end='', flush=True)
    if chunk.message.tool_calls:
        print(chunk.message.tool_calls)

Output


[ToolCall(function=Function(name='add_two_numbers', arguments={'a': 3, 'b': 1}))]

JavaScript SDK usage

The JavaScript client (npm i ollama) works similarly. A tool schema is defined, then ollama.chat() streams both text and tool calls.

import ollama from 'ollama';

const addTool = {
  type: 'function',
  function: {
    name: 'addTwoNumbers',
    description: 'Add two numbers together',
    parameters: {
      type: 'object',
      required: ['a', 'b'],
      properties: {
        a: {type: 'number', description: 'The first number'},
        b: {type: 'number', description: 'The second number'}
      }
    }
  }
};

async function run(model) {
  const messages = [{role: 'user', content: 'What is 2 plus 3?'}];
  for await (const chunk of await ollama.chat({model, messages, tools: [addTool], stream: true})) {
    if (chunk.message.tool_calls) {
      console.log('Tool call:', chunk.message.tool_calls);
    } else {
      process.stdout.write(chunk.message.content);
    }
  }
}

run('qwen3').catch(console.error);

Output

Question: What is 2 plus 3?

Tool call: {function: {name: "addTwoNumbers", arguments: {a: 2, b: 3}}}

How the incremental parser works

Background

Earlier Ollama versions buffered the entire model output, parsed it as JSON, and then emitted tool calls. This blocked streaming because a tool call could appear anywhere in the text.

Incremental parsing strategy

The new parser reads each model’s template to recognize tool‑call prefixes (e.g., special tokens or strings). It can:

Detect a partial prefix as the model streams tokens.
Separate regular content from a pending tool call.
Fall back to generic JSON detection when a model emits a raw JSON object without the expected prefix.

Handling edge cases

If a model repeats a previously made tool call or includes the call inside explanatory text, the parser’s state machine avoids duplicate calls. Empirical tests show that malformed or duplicated calls are now reduced to a single, correct invocation.

Accuracy improvements

Previously, a model could produce two identical tool calls when it reiterated the call in its narrative. The updated parser matches prefixes and tracks JSON parsing state, ensuring only one call is emitted. Example of the old failure:

[TOOL_CALL] [{"name":"get_conditions","arguments":{"city":"Sydney"}}]
... (text) ...
[{"name":"get_conditions","arguments":{"city":"Sydney"}}]

Now the parser returns a single tool_calls entry.

Model Context Protocol (MCP) and larger windows

Streaming tool calls work with Ollama’s Model Context Protocol. Users report that increasing the context window to 32 k tokens or more improves both the reliability of tool detection and the quality of the generated answer.

Adjusting the context window via cURL

curl -X POST "http://localhost:11434/api/chat" -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "why is the sky blue?"}],
  "options": {"num_ctx": 32000}
}'

Note: larger windows increase memory consumption.

Getting started

Download the latest Ollama release – https://ollama.com/download
Install the language‑specific SDKs – pip install -U ollama for Python, npm i ollama for JavaScript.
Use the stream: true flag in API calls and include a tools array describing the functions you want the model to call.

Reference

Pull request implementing streaming tool calls: https://github.com/ollama/ollama/pull/10415