grafana/promql-anomaly-detection

What it solves

This framework provides a way to detect anomalies in time series data directly within Prometheus, removing the need for external machine learning systems or complex data pipelines. It helps identify unusual behavior in metrics like request rates, latency, error rates, and resource usage by automatically calculating expected ranges.

How it works

The framework uses Prometheus recording rules to calculate "anomaly bands"—upper and lower bounds based on historical data. It supports two primary strategies: an adaptive approach (using mean and standard deviation) for quick detection of short-term changes, and a robust approach (using median and median absolute deviation) for handling spiky or non-normally distributed data. Users tag their metrics with specific labels (anomaly_name, anomaly_strategy, and anomaly_type) to trigger the detection logic, which then generates alerts if the actual metric values cross these calculated bands.

Who it’s for

This is for DevOps engineers, SREs, and system administrators who already use Prometheus and Grafana for monitoring and want to implement automated anomaly detection without introducing external dependencies or specialized AI/ML infrastructure.

Highlights

• Runs entirely within Prometheus using standard PromQL, requiring no external services.
• Supports multiple detection algorithms tailored for different metric types like latency or error rates.
• Includes pre-built Grafana dashboard templates for visualizing anomaly bands.
• Highly extensible, allowing users to define custom recording and alerting rules based on their specific infrastructure needs.