antoinezambelli/forge

A Python framework for self-hosted LLM tool-calling and multi-step agentic workflows

What it solves

Forge provides a reliability layer for self-hosted LLM tool-calling. It addresses the instability of local models (especially smaller ones around 8B parameters) when executing tools, which often suffer from malformed output, incorrect tool selection, or failure to follow structured workflows.

How it works

Forge implements a stack of guardrails that sit between the LLM and the application. It can be used in three ways:

  1. Proxy Server: A drop-in proxy that speaks OpenAI and Anthropic APIs. It intercepts requests and applies guardrails transparently, allowing existing tools (like aider or Claude Code) to work with local models without code changes.
  2. WorkflowRunner: A high-level framework for defining tools and structured agent loops, managing the full lifecycle from system prompts to context compaction.
  3. Guardrails Middleware: A composable set of tools that can be integrated into existing custom orchestration loops.

Key technical mechanisms include:

  • Rescue Parsing: Extracts structured tool calls from non-canonical formats (e.g., XML or fenced JSON) and converts them to standard schemas.
  • Response Validation: Checks tool calls against defined specifications before they reach the client.
  • Retry Loops: Automatically retries failed inferences with corrective "nudges" to the model.
  • Synthetic respond Tool: Forces models to use a specific tool for text responses, preventing the common confusion between generating text and calling a tool.

Who it’s for

Developers building agentic applications with self-hosted LLMs (via Ollama, vLLM, llama.cpp, etc.) who need production-grade reliability for tool-calling without switching to massive frontier models.

Highlights

  • Significant Performance Boost: Lifts 8B local models from single-digit reliability to 84% on its eval suite.
  • Broad Backend Support: Compatible with Ollama, llama-server, Llamafile, vLLM, and Anthropic.
  • Zero-Rewrite Integration: The proxy mode allows it to fortify existing AI coding harnesses without modifying their source code.
  • Context Management: Includes tiered compaction strategies to manage token budgets efficiently.