Nightwatch: An Open-Source, Read-Only AI SRE
Nightwatch is an open-source, local-first AI SRE layer designed to reduce alert fatigue and automate root-cause analysis. It transforms "alert storms"—where a single outage causes dozens of disparate alerts—into single, consolidated incidents, and utilizes a tool-calling AI agent to investigate live systems and propose fixes that must be approved by a human operator.
Read-Only Architecture by Design
Nightwatch is strictly read-only. It observes, reasons, and recommends, but never executes commands, acknowledges alerts, or modifies production environments. Every proposed fix is provided as a copy-pasteable artifact for a human to gate and execute. This design ensures that the AI does not introduce new failures into a live system during an investigation.
The AI SRE Investigator
At the core of Nightwatch is an AI SRE investigator that uses a ReAct loop (reason $\rightarrow$ act $\rightarrow$ observe) to gather evidence from live systems. The agent is driven by a typed allowlist of read-only capabilities, ensuring it can only perform read operations.
Investigation Capabilities
| Capability | Read-Only Access |
|---|---|
| Docker | Containers, logs, stats, and inspection |
| Kubernetes | Pods, logs, events, and deployments via in-cluster RBAC |
| AWS | CloudTrail change events, EC2, security groups, and quotas via IAM read-role |
| Grafana | PromQL and LogQL via a datasource proxy |
| GitHub | CI runs, releases, and PRs for change-event RCA |
| Git | Mirrored repositories for commits, diffs, and code search |
| Host | CPU, memory, disk, processes, and socket logs for plain VMs |
Safety and Grounding
To prevent hallucinations and security risks, Nightwatch employs several hardening measures:
- Injection Shielding: Untrusted logs and diffs are shielded to prevent prompt injection.
- Secret Scrubbing: Credentials and sensitive data are one-way scrubbed before being sent to LLM providers.
- Grounding Gate: A mechanism that caps confidence levels when claims are not backed by evidence.
- Action Classification: Every action is classified as
read_only,reversible, orirreversible. Unknown actions are coerced toirreversibleby default.
Distributed Investigation via Ninox Runners
Nightwatch utilizes a "brain" and "runner" architecture to investigate systems in restricted environments. A ninox runner is a thin, outbound-only agent that lives inside a VPC, cluster, or on-prem segment. The runner holds credentials locally and dials home to the brain, eliminating the need for inbound firewall holes.
Alert Clustering and Noise Reduction
Nightwatch sits above existing monitoring tools (such as Prometheus Alertmanager, Checkmk, Icinga2, and Zabbix) and normalizes alerts into a single schema. It then clusters alerts by host, service, severity, and time-window to group them into single incidents.
The system also identifies "noisy" checks—those that are flapping or over-sensitive—and provides rule-based tuning recommendations with evidence to provide a rationale for the human operator.
LLM Integration and Privacy
Nightwatch supports multiple LLM providers, including Anthropic (default for the investigator), OpenAI, Mistral, and local LLMs via Ollama or vLLM. It also includes a template provider that works fully offline for basic summaries and recommendations without requiring an LLM.
To maintain privacy, redaction and secret-scrubbing run before every remote call, replacing hostnames, IPs, and UUIDs with deterministic placeholders.
Community and Extension
Nightwatch is licensed under the Apache License 2.0. Users can extend the AI SRE's capabilities by pointing it at an MCP server, writing a Python capability plugin, or using the runner protocol.