messkan/prompt-cache

Cut LLM costs by up to 80% and unlock sub-millisecond responses with intelligent semantic caching.A drop-in, provider-agnostic LLM proxy written in Go with sub-millisecond response

What it solves

PromptCache reduces the cost and latency of LLM applications by avoiding redundant upstream API calls. It addresses the problem of repeated or near-repeated requests in workloads like RAG applications, AI agents, and support bots where similar questions often receive the same answer.

How it works

It acts as a self-hosted proxy between an application and an LLM provider. It uses a two-threshold semantic matching strategy to determine if a new prompt is similar enough to a cached response:

  1. High similarity: Results in a direct cache hit.
  2. Low similarity: Results in a cache miss.
  3. Gray zone: Optionally uses a smaller model to verify the intent of the prompt before serving a cached response.

It is OpenAI-compatible, meaning it can be used with existing SDKs by simply changing the base URL. It persists data using BadgerDB and supports streaming responses via SSE.

Who it’s for

Developers building production GenAI workloads who want to lower provider costs, reduce response times, and increase their rate-limit headroom.

Highlights

  • OpenAI-compatible API: Seamlessly integrates with existing OpenAI-compatible clients.
  • Semantic Matching: Uses embeddings to find similar prompts rather than exact string matches.
  • Multi-provider support: Works with OpenAI, Mistral AI, and Anthropic/Claude (via Voyage AI).
  • Runtime Configuration: Allows updating similarity thresholds and providers without restarting the service.
  • Cache Warming: Supports pre-populating the cache with known prompt-response pairs.
  • Observability: Includes Prometheus metrics and structured logging.

Related

  • Dispatch
    OpenAI Prompt Caching API Release

    OpenAI has introduced Prompt Caching for GPT-4o, GPT-4o mini, o1-preview, and o1-mini, offering a 50% discount and reduced latency for reused input tokens.

  • Project
    arakoodev/EdgeChains

    A GenAI configuration framework that uses Jsonnet to manage prompts and chains as declarative configuration, enabling versioning and automatic hardware parallelism via WebAssembly.

  • Project
    larsderidder/context-lens

    A local proxy that captures LLM API calls from AI coding tools to provide a detailed visual breakdown of context window composition and token costs.

  • Project
    jia-gao/leanctx

    A drop-in prompt compression library that reduces LLM input-token costs by 10-40% using loss-tolerance routing to protect critical code and tool calls while compressing prose.

  • Project
    MagnivOrg/prompt-layer-library

    A Python SDK for PromptLayer that enables prompt versioning, tracing, and monitoring for LLM applications and agents across multiple AI providers.