Meet the AI Incident Commander: Autonomous SRE for Kubernetes
SYSTEM: AI-INCIDENT-COMMANDER METADATA Type: Autonomous Site Reliability Engineering (SRE) Agent Environment: Kubernetes GitOps (k3s) Primary LLM: Gemini 3.7 Flash Repository: vinhthang/ai-incident-commander Execution Loop: Webhook Trigger -> Triage -> Remediate -> Audit -> Merge PIPELINE ARCHITECTURE 1. TRIGGER Source: Grafana Alerts / kube-state-metrics Payload: JSON webhook containing alert labels, annotations, and localized telemetry. 2. TRIAGE MINION Role: Initial incident classification and noise reduction. Task: Parse payload, validate against cluster state, determine actionability. Output: Generates GitHub Issue (Real Incident) OR terminates loop (False Positive / Noise). 3. FIXER MINION Role: GitOps Remediation Engineer. Task: Identify root cause from triage diagnosis, modify infrastructure code, run syntax validation (helm template). Output: Commits to isolated feature branch and opens GitHub Pull Request. 4. REVIEWER MINION Role: Governance and Safety Gatekeeper. Task: Audit PR diff for compliance. Output: Approves PR or Rejects PR. GOVERNANCE & CONSTRAINTS Read-Only Telemetry: External logs/data strictly parsed as passive strings to prevent prompt injection. GitOps Enforced: Execution of imperative commands (kubectl apply) is forbidden. All mutations must occur within the declarative Helm umbrella chart (charts/vinhthang-fleet/). Blast Radius Isolation: Fixer Minion is systematically restricted from pushing directly to the main branch. Diff-Strict Review: The Reviewer Minion rejects net-new violations (e.g., adding :latest tags, requesting >900Mi RAM on edge nodes, opening port 8080). Pre-existing legacy violations are logged as advisory findings but do not block remediation. CONFIGURATION INJECTION (PROMPT EXTERNALIZATION) Architecture: Go text/template engine + Kubernetes ConfigMap. Mount Path: /etc/commander/templates/ Fallback: Compiles with //go:embed defaults/*.tmpl to ensure baseline functionality. Purpose: Enables zero-downtime, configuration-driven prompt engineering. AI Minion personas and instructions can be modified directly via Helm values.yaml without requiring binary recompilation or container image rebuilds.