sev0

Problem

Diagnosing a production incident means reading metrics, logs, and git history to find a root cause, then verifying a fix actually works before anyone trusts it — a slow, manual process even for an experienced on-call engineer.

Challenges

• Orchestrating 16 diagnostic tools across Prometheus, Loki, Git, and Docker to trace a failure to its root-cause service, file, symbol, and introducing commit
• Reproducing a failure before trusting a patch that claims to fix it, rather than accepting green tests as sufficient
• Building a 4-service incident lab with 23 fault scenarios — including adversarial and silent failures — to actually evaluate root-cause accuracy
• Enforcing sandbox isolation, patch size limits, and protected paths in code rather than in the prompt

Solution

An autonomous AI software engineer that reads metrics and logs to find what broke and when, reads git history around that moment, forms and tests hypotheses by executing code inside an isolated Docker sandbox, and only proposes a patch after reproducing the failure it claims to fix. A verified fix opens as a draft pull request — never merged automatically.

Architecture

A hand-rolled tool-calling loop against the Anthropic Messages API drives 16 tools across four stages: collect (Prometheus/Loki metrics and logs), retrieve (git history and symbol-level code reading), experiment (hypotheses tested by execution in a networkless Docker sandbox), and repair (patch applied to a throwaway copy, verified against the real test suite, opened as a draft PR only if the failure reproduced and the fix held). Incident Lab — a real 4-service storefront with 23 fault scenarios — provides ground truth the agent can't read, used to benchmark diagnostic accuracy across ablations.

Stack

Highlights

• 16 diagnostic tools across observability, git history, code retrieval, execution, and reasoning
• 4-service Incident Lab with 23 fault scenarios, including adversarial and silent-failure cases
• Reproduces failures before trusting a patch, and reports regressions separately from fixes
• Sandboxed execution, patch size limits, and protected paths enforced in code — draft PRs only, nothing merges automatically