Ollama and Continue: Setting Up an Open-Source AI Code Assistant

Ollama and Continue provide a framework for creating a fully open-source AI coding assistant that runs locally on a laptop or a remote server. This integration allows developers to maintain control over their code and data while accessing advanced autocomplete and chat capabilities directly within Visual Studio Code and JetBrains editors.

Local Model Configurations for Chat and Autocomplete

Developers can configure different LLMs to handle specific coding tasks based on their available hardware resources (VRAM).

High-Performance Single Model: Codestral 22B

Mistral AI’s Codestral 22B is recommended for users who want a single model to handle both autocomplete and chat. While highly capable for programming tasks, it requires significant VRAM and is subject to a non-production license, limiting its use to research and testing.

Optimized Multi-Model Setup: DeepSeek Coder and Llama 3

For users with limited VRAM or those who prefer specialized models, a split configuration is possible:

Autocomplete: DeepSeek Coder 6.7B is recommended for efficient code completion.
Chat: Llama 3 8B is recommended for the conversational interface.

Ollama's ability to handle multiple concurrent requests allows these models to run simultaneously if the hardware supports it. If not, users can combine local models with remotely hosted or SaaS providers.

Local Codebase Context and Retrieval

Continue integrates with Ollama to provide local RAG (Retrieval-Augmented Generation) capabilities, ensuring that codebase context remains private.

Codebase Indexing with @codebase

The @codebase context provider allows the assistant to retrieve relevant snippets from the local project. This is powered by nomic-embed-text embeddings running via Ollama and indexed using LanceDB. This enables developers to ask project-specific questions, such as querying the default context length of a specific model within their own code.

Documentation Integration with @docs

The @docs provider allows users to index external documentation sites. By providing a URL (such as the Ollama GitHub README), the assistant can retrieve and use specific documentation sections as context for chat queries, keeping the retrieval process local.

Model Fine-Tuning and Customization

Continue allows teams to improve their AI assistants by leveraging their own development data.

Fine-Tuning StarCoder 2

Continue automatically saves development data, including accepted autocomplete suggestions, to .continue/dev_data. This data can be used to fine-tune models like StarCoder 2. The recommended workflow involves:

Extracting accepted tab suggestions into Hugging Face Datasets.
Using the Hugging Face Supervised Fine-tuning Trainer.
Pushing the resulting fine-tuned model to the Ollama model library for team-wide deployment and acceptance rate measurement.