prehisle/relay-pulse

What it solves

Relay Pulse addresses the problem of "fake availability" in LLM API proxy services. Traditional monitoring tools only check if an HTTP connection is successful (HTTP 200), but an LLM service might return a 200 status code while delivering empty content or internal error messages. This tool ensures a service is actually functional by verifying that the LLM generates real text.

How it works

The system performs periodic "real-world" probes by sending actual API requests to configured LLM providers. It consumes a small number of real tokens (typically using max_tokens: 1) to verify that the model actually "speaks." It tracks these results over time and stores them in either SQLite or PostgreSQL, providing a visual dashboard with availability heatmaps for 24-hour, 7-day, and 30-day windows.

Who it’s for

• Users who self-host or purchase LLM proxy/relay services and need to track uptime.
• Developers comparing the quality, latency, and error rates of multiple cloud LLM providers.
• Businesses relying on external LLM APIs who want to avoid service outages caused by "fake" successful responses.

Highlights

• Real Token Probing: Validates actual output generation rather than just network connectivity.
• Visual Quality Matrix: Provides availability heatmaps to identify service stability trends.
• Hot Configuration Updates: Uses fsnotify to apply configuration changes without requiring a server restart.
• Cloud-Native Design: Supports Docker deployment and horizontal scaling via PostgreSQL.
• Low Overhead: Designed for minimal cost, consuming roughly 30,000 tokens per day per service.