huggingface/llm.nvim

llm.nvim – LLM‑powered code completion for Neovim

What it is

• A Neovim plugin that lets you get AI‑generated code suggestions directly inside the editor, similar to GitHub Copilot.
• It talks to a small language‑server binary called llm‑ls, which in turn forwards HTTP requests to an LLM backend (Hugging Face Inference API, Ollama, OpenAI‑compatible servers, or Hugging Face Text‑Generation‑Inference).

Key features

Feature	What you get
Ghost‑text completion	Inline “ghost” suggestions appear as you type, just like Copilot’s UI.
Model choice	Pick any model hosted on Hugging Face, an Ollama instance, an OpenAI‑compatible endpoint, or a TGI server.
Context‑window safety	The plugin trims the prompt so it never exceeds the model’s token limit, using the tokenizers library to count tokens accurately.
Fill‑in‑the‑middle (FIM) support	For models that understand FIM (e.g., StarCoder, CodeLlama) you can enable prefix/middle/suffix markers.
Per‑file enable/disable	Configure suggestion activation with glob patterns (e.g., *.py, */my_project/*).
Auto‑suggest toggle	LLMToggleAutoSuggest lets you turn the live‑suggestion stream on or off at runtime.
Manual request	LLMSuggestion forces a single completion request.
Flexible request body	Any JSON fields you add under request_body are passed straight to the backend, so you can tweak temperature, top‑p, max tokens, etc.

How it works

1. Setup – When the plugin is first loaded it downloads the llm‑ls binary from its GitHub releases (or you can install it via mason.nvim).
2. Backend selection – You tell the plugin which backend to use (backend = "huggingface" | "ollama" | "openai" | "tgi"). The plugin builds the correct URL and request payload for that service.
3. Prompt preparation – Your current buffer, plus optional FIM markers, is tokenized. If the token count would exceed the model’s context_window, the oldest tokens are dropped so the request fits.
4. HTTP request – An HTTP POST is sent to the backend (or to a local Ollama/TGI server). If an api_token is supplied it is added as a Bearer header.
5. Response handling – The raw text returned by the model is cleaned (tokens in tokens_to_clear are stripped) and displayed as ghost‑text. Accepting (<Tab> by default) inserts the suggestion; <S‑Tab> dismisses it.

Installation

-- using lazy.nvim (recommended)
require('lazy').setup({
  {
    'huggingface/llm.nvim',
    opts = {
      -- minimal example configuration
      backend = 'huggingface',
      model   = 'bigcode/starcoder',
    },
  },
})

You can also install with packer.nvim or vim-plug; the README provides the exact snippets.

Basic configuration (Lua)

require('llm').setup({
  api_token = nil,               -- token for the chosen backend (or set via env vars)
  model     = 'bigcode/starcoder',
  backend   = 'huggingface',
  request_body = {
    parameters = {
      max_new_tokens = 60,
      temperature    = 0.2,
      top_p          = 0.95,
    },
  },
  fim = { enabled = true, prefix = '<fim_prefix>', middle = '<fim_middle>', suffix = '<fim_suffix>' },
  enable_suggestions_on_startup = true,
  enable_suggestions_on_files   = '*',
})

All options are optional; defaults are documented in the README (e.g., StarCoder defaults, tokenizers, debounce time, keymaps).

When to use it

• You already work in Neovim and want AI‑assisted coding without leaving the editor.
• You prefer self‑hosted or open‑source models (Ollama, TGI) or want to use Hugging Face’s hosted inference.
• You need fine‑grained control over which files get suggestions and how the request payload looks.

Limitations / gotchas

• The Hugging  Face Inference API free tier is rate‑limited; the README advises a PRO subscription for heavy use.
• The plugin only supports text‑generation style models; it won’t work with chat‑oriented APIs unless they expose a compatible /generate endpoint.
• Context‑window size is limited by the chosen model (e.g., 8 k tokens for StarCoder). The plugin truncates older context automatically, which may affect suggestions in very large files.
• You must provide a valid API token or run a local server; otherwise the HTTP request will fail.

Where to learn more

• The plugin’s README (the source you’re reading) contains full option tables and examples.
• llm‑ls repository explains how the language‑server binary is built and how to run it manually.
• Hugging  Face’s Inference API docs, Ollama API docs, and the TGI docs are linked for each backend.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.