agentjido/req_llm

What is ReqLLM?

ReqLLM is an open‑source Elixir library that lets you talk to many large‑language‑model (LLM) providers (OpenAI, Anthropic, Azure, Google Gemini, etc.) through a single, consistent API. It builds on the popular Req HTTP client and Finch streaming library, handling the quirks of each provider so you can write the same Elixir code no matter which model you’re using.

Why it matters

Provider‑agnostic – 1 200+ models from 21+ providers are registered in the library’s model catalog. You pick a model with a string like "anthropic:claude-haiku-4-5" and the library translates the request into the right HTTP shape for that provider.
Unified high‑level helpers – Functions such as generate_text/3, stream_text/3, generate_object/4 mimic the Vercel AI SDK style, so you can generate plain text, structured JSON, or even images with the same call signature.
Streaming support – Real‑time token streaming works across providers via Finch, giving you a StreamResponse that yields tokens while still collecting usage metadata.
Structured output & tool calling – You can describe the shape of the expected result (object, array, JSON schema, etc.) and the library will validate the provider’s response, making function‑calling workflows easier.
Cost & usage tracking – Every response includes token counts and a best‑effort USD cost estimate, plus telemetry events that can be hooked into OpenTelemetry or other observability stacks.

Core concepts (quick cheat‑sheet)

Concept	What it is	Typical use
Model spec	A string, tuple, or `%LLMDB.Model{}` that identifies a provider and model ID (e.g. `"openai:gpt-4o"`).	Choose which model to call.
Context	A list of system/user/assistant messages, built with `ReqLLM.Context.*`.	Provide multi‑turn conversation history.
Output descriptor	`ReqLLM.Output` structs that declare the expected shape (text, object, array, JSON schema, etc.).	Ask the model to return structured data.
Response structs	`ReqLLM.Response`, `ReqLLM.StreamResponse`, `ReqLLM.Usage`.	Inspect the generated text, images, embeddings, usage, and cost.
Provider transports	Internally, Req builds the HTTP request; Finch handles streaming.	You never call Req or Finch directly.
Key management	`ReqLLM.Keys` pulls API keys from env vars, `.env` files, or in‑memory storage.	Keep credentials out of code.

Getting started (the shortest path)

# Add the library via Igniter (recommended)
mix igniter.install req_llm

# In your code
model = "anthropic:claude-haiku-4-5"

# Simple one‑shot text generation
text = ReqLLM.generate_text!(model, "Hello world")
# => "Hello! How can I assist you today?"

# Structured object generation
schema = [name: [type: :string, required: true], age: [type: :pos_integer]]
person = ReqLLM.generate_object!(model, "Generate a person", schema)
# => %{name: "John Doe", age: 30}

For streaming:

{:ok, resp} = ReqLLM.stream_text(model, "Write a short story")
ReqLLM.StreamResponse.tokens(resp) |> Stream.each(&IO.write/1) |> Stream.run()

Notable features worth a deeper look

Provider‑specific options – e.g., provider_options: [web_search: %{max_uses: 5}] enables Anthropic’s web‑search tool.
Embedding generation – Embedding.generate/3 returns single or batch embeddings where the provider supports it.
Realtime OpenAI sessions – Low‑level WebSocket API (ReqLLM.OpenAI.Realtime) for the newer realtime models.
Telemetry – Built‑in Telemetry events ([:req_llm, :request, …], [:req_llm, :token_usage]) and optional OpenTelemetry attachment for full tracing.
Extensible model registry – You can call models not yet in the catalog by passing a full spec map and normalising it with ReqLLM.model!/1.

Who should use this?

Elixir developers building chatbots, agents, or any GenAI‑powered service who want a single, idiomatic client instead of juggling dozens of provider SDKs.
Teams that need cost visibility and structured‑output validation out of the box.
Ops / observability folks who want ready‑made Telemetry hooks for billing dashboards.

Where to learn more

Hex package page – https://hex.pm/packages/req_llm
Full documentation – https://hexdocs.pm/req_llm/
Provider guides – see guides/ in the repo (e.g., guides/openai.md).
Discord community – https://jido.run/discord (The Swarm: Elixir AI Collective).

TL;DR

ReqLLM gives you a single Elixir API to call hundreds of LLM models (text, embeddings, images, speech, OCR, etc.), with streaming, structured output, cost tracking, and telemetry baked in. It abstracts away each provider’s idiosyncrasies, letting you focus on the prompt and the data you need.