graniet/llm

A powerful Rust library and CLI tool to unify and orchestrate multiple LLM, Agent and voice backends (OpenAI, Claude, Gemini, Ollama, ElevenLabs...) with a single, extensible API. Build, chain, evaluate, and serve complex multi-step AI workflows — including speech-to-text, text-to-speech, completions, vision, and reasoning.

What it solves

It provides a unified Rust library to interact with multiple LLM providers (such as OpenAI, Anthropic, Google, and Ollama) without needing to write provider-specific code for each one. It simplifies the process of switching between models or combining different backends into a single workflow.

How it works

The library uses a builder pattern and unified traits (ChatProvider and CompletionProvider) to standardize how requests are sent to various AI backends. It allows developers to configure models, temperature, and tokens through a single API. It also supports creating multi-step chains where different backends can be used at each step, and can serve these chains as a REST API in the OpenAI standard format.

Who it’s for

Rust developers who want to integrate multiple AI models into their applications while maintaining a clean, provider-agnostic codebase.

Highlights

  • Multi-backend support: Works with OpenAI, Anthropic, Ollama, DeepSeek, xAI, Phind, Groq, Google, Cohere, Mistral, Hugging Face, and ElevenLabs.
  • Advanced Orchestration: Supports multi-step chains, reactive agents with shared memory, and parallel evaluation of multiple providers to select the best response.
  • Rich Feature Set: Includes built-in support for vision, reasoning, structured JSON output, and speech-to-text/text-to-speech.
  • Developer Tooling: Includes a command-line interface (CLI) for quick model interaction and resilience features like exponential backoff and jitter.