grafana/promql-anomaly-detection

What it solves

This framework provides a way to detect anomalies in time series data directly within Prometheus using PromQL, eliminating the need for external anomaly detection systems.

How it works

The system uses a set of recording rules to create upper and lower "anomaly bands" around a metric. These bands are calculated using different strategies:

• Adaptive Strategy: Uses mean and standard deviation with a smoothing function and high-pass filter to detect short-term changes while reducing false positives for recurring events.
• Robust Strategy: Uses median and median absolute deviation (MAD) to handle spiky or non-normally distributed data and detect long-term changes.

These bands combine short-term variability, seasonality (daily or weekly patterns), and margin bands to ensure a minimum width. Alerting rules then trigger when a metric crosses these bands for a significant duration.

Who it’s for

Users of Prometheus and Grafana who want to implement automated anomaly detection for metrics like request rates, latency, error rates, and resource usage without adding new infrastructure.

Highlights

• No External Dependencies: Works entirely within the Prometheus/PromQL ecosystem.
• Seasonality Support: Adapts to recurring daily or weekly patterns.
• Flexible Strategies: Offers both adaptive and robust algorithms depending on the data distribution.
• Grafana Integration: Supports overlaying anomaly bands directly on time series panels for visualization.