jeremychone/rust-genai

Overview

genai is a Rust library that gives developers a single, ergonomic API for talking to many large‑language‑model (LLM) providers. It abstracts the differences between providers (OpenAI, Anthropic, Gemini, Ollama, Bedrock, etc.) while still letting you use each provider’s native protocol when it exists. In practice you create a Client, build a ChatRequest with system/user messages, and call exec_chat (or the streaming variant) with a model name such as gpt-5.6-luna or a namespaced name like qwen_cloud::qwen3.8-max.

---

Key Features

Feature	What it means for you
Multi‑provider support	27+ providers and 200+ models are built‑in. The library picks the right adapter from the model name, or you can force one with a namespace::model syntax.
Native‑protocol adapters	When a provider offers a proprietary API (e.g., Anthropic, Gemini, Bedrock) genai talks to it directly, preserving advanced options like reasoning budgets, tool‑call metadata, and streaming frames.
OpenAI‑compatible fallback	For services that expose an OpenAI‑style endpoint (Ollama, custom self‑hosted servers, Groq, etc.) genai uses a generic OpenAI‑compatible adapter.
Streaming & frame inspection	exec_chat_stream returns a stream of ChatFrames; you can attach a ChatFrameSink to look at raw frames for debugging or custom processing.
Tool / function calling	Provider‑neutral ToolChoice and built‑in tool structs (e.g., WebSearch) let you use tool calls on Anthropic, OpenAI, Gemini, etc., without writing provider‑specific code.
Reasoning effort control	Enums like ReasoningEffort::High map to provider‑specific knobs (e.g., Anthropic’s max_tokens or OpenAI’s temperature).
Image analysis	Simple helper for providers that accept images (OpenAI Vision, Gemini Flash‑2, Anthropic).
Custom endpoints & auth	Set environment variables (GENAI_1_ENDPOINT, GENAI_1_API_KEY, OMLX_ENDPOINT, …) or implement ServiceTargetResolver to point the client at any OpenAI‑compatible server.
TLS backend selection	Choose between native‑tls, rustls, or OpenSSL at compile time.
OpenTelemetry integration	Optional otel feature adds GenAI‑semantic‑convention tracing.

---

How It Works

1. Client creation – let client = Client::new()?; (now fallible, returns Result).
2. Model resolution – The library infers an AdapterKind from the model prefix (gpt‑* → OpenAI, claude‑* → Anthropic, etc.). You can override with namespace::model. The adapter knows the correct HTTP endpoint, auth header, and request shape.
3. Chat request – Build a ChatRequest with a vector of ChatMessage (system, user, assistant). Optional ChatOptions let you set temperature, max tokens, tool choice, extra body fields, etc.
4. Execution – Call client.exec_chat(model, request, None).await? for a single response or exec_chat_stream for a streaming response.
5. Result handling – The returned ChatResponse gives you the generated text (first_text()), tool call data, and raw streaming frames if you need them.

---

Example (simplified)

use genai::{Client, chat::{ChatMessage, ChatRequest}};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1️⃣ Create the client (reads env vars for API keys)
    let client = Client::new()?;

    // 2️⃣ Build a simple chat request
    let req = ChatRequest::new(vec![
        ChatMessage::system("You are a concise assistant"),
        ChatMessage::user("Why is the sky red?"),
    ]);

    // 3️⃣ Call any model – here an OpenAI‑compatible one
    let resp = client.exec_chat("gpt-5.6-luna", req, None).await?;
    println!("Answer: {}", resp.first_text().unwrap_or("<none>"));
    Ok(())
}

---

When to Use genai

• You need to switch between providers (e.g., test a prompt on OpenAI, Anthropic, and a local Ollama model) without rewriting request code.
• You want a single Rust crate instead of pulling in many provider‑specific SDKs.
• You care about provider‑specific features such as Anthropic’s tool caching, Gemini’s structured output, or Bedrock’s signed reasoning blocks – genai exposes them while keeping a common surface.
• You are building a service or CLI that may run locally (Ollama) for dev and in the cloud (OpenAI, Bedrock) in production.

---

Limitations / Things to Keep in Mind

Aspect	Note
Depth of API	The crate focuses on chat/completion, streaming, and a few multimodal helpers. Full‑featured SDKs (e.g., async‑openai) may have more endpoints (embeddings, fine‑tuning, moderation).
Provider‑specific quirks	While most knobs are unified, some advanced features (e.g., OpenAI’s logprobs, Azure resource‑specific headers) are not yet exposed.
Async runtime	Requires Tokio (or any runtime that implements AsyncRead/Write).
Version cadence	Rapid releases (0.6 → 0.7) may introduce breaking changes; consult the migration guide when upgrading.

---

Getting Started

1. Add to Cargo.toml:

   genai = "0.6"
   

2. Set the appropriate environment variables for the providers you plan to use (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY).
3. Run one of the bundled examples, e.g., cargo run --example c00-readme.

---

Community & Contributions

• Crates.io –
• GitHub – Issues, PRs, and a BIG-THANKS.md list of contributors.
• Video tutorials – A YouTube playlist walks through setup, model mapping, and adding new providers.
• Feature flags – Choose TLS backend (native-tls, rustls-tls, openssl-tls) and optional OpenTelemetry (otel).

---

TL;DR

genai is a Rust‑first, provider‑agnostic wrapper for LLM chat APIs. Write one piece of Rust code, point it at any of dozens of models—from OpenAI’s GPT‑4o to a locally‑run Ollama model—and get back consistent responses while still being able to tap into each service’s unique capabilities.