https://vinhthang.dev/tags/incident-response/Recent content in Incident Response on Vinh Thang's AI Tech NotesHugoen-usMon, 31 Aug 2026 00:43:00 +0700https://vinhthang.dev/posts/ai-incident-commander/Mon, 31 Aug 2026 00:43:00 +0700https://vinhthang.dev/posts/ai-incident-commander/<h1 id="system-ai-incident-commander">SYSTEM: AI-INCIDENT-COMMANDER</h1> <h2 id="metadata">METADATA</h2> <ul> <li><strong>Type</strong>: Autonomous Site Reliability Engineering (SRE) Agent</li> <li><strong>Environment</strong>: Kubernetes GitOps (k3s)</li> <li><strong>Primary LLM</strong>: Gemini 3.7 Flash</li> <li><strong>Repository</strong>: <a href="https://github.com/vinhthang/ai-incident-commander">vinhthang/ai-incident-commander</a></li> <li><strong>Execution Loop</strong>: Webhook Trigger -&gt; Triage -&gt; Remediate -&gt; Audit -&gt; Merge</li> </ul> <h2 id="pipeline-architecture">PIPELINE ARCHITECTURE</h2> <h3 id="1-trigger">1. TRIGGER</h3> <ul> <li><strong>Source</strong>: Grafana Alerts / kube-state-metrics</li> <li><strong>Payload</strong>: JSON webhook containing alert labels, annotations, and localized telemetry.</li> </ul> <h3 id="2-triage-minion">2. TRIAGE MINION</h3> <ul> <li><strong>Role</strong>: Initial incident classification and noise reduction.</li> <li><strong>Task</strong>: Parse payload, validate against cluster state, determine actionability.</li> <li><strong>Output</strong>: Generates GitHub Issue (Real Incident) OR terminates loop (False Positive / Noise).</li> </ul> <h3 id="3-fixer-minion">3. FIXER MINION</h3> <ul> <li><strong>Role</strong>: GitOps Remediation Engineer.</li> <li><strong>Task</strong>: Identify root cause from triage diagnosis, modify infrastructure code, run syntax validation (<code>helm template</code>).</li> <li><strong>Output</strong>: Commits to isolated feature branch and opens GitHub Pull Request.</li> </ul> <h3 id="4-reviewer-minion">4. REVIEWER MINION</h3> <ul> <li><strong>Role</strong>: Governance and Safety Gatekeeper.</li> <li><strong>Task</strong>: Audit PR diff for compliance.</li> <li><strong>Output</strong>: Approves PR or Rejects PR.</li> </ul> <h2 id="governance--constraints">GOVERNANCE &amp; CONSTRAINTS</h2> <ul> <li><strong>Read-Only Telemetry</strong>: External logs/data strictly parsed as passive strings to prevent prompt injection.</li> <li><strong>GitOps Enforced</strong>: Execution of imperative commands (<code>kubectl apply</code>) is forbidden. All mutations must occur within the declarative Helm umbrella chart (<code>charts/vinhthang-fleet/</code>).</li> <li><strong>Blast Radius Isolation</strong>: Fixer Minion is systematically restricted from pushing directly to the <code>main</code> branch.</li> <li><strong>Diff-Strict Review</strong>: The Reviewer Minion rejects net-new violations (e.g., adding <code>:latest</code> tags, requesting &gt;900Mi RAM on edge nodes, opening port 8080). Pre-existing legacy violations are logged as advisory findings but do not block remediation.</li> </ul> <h2 id="configuration-injection-prompt-externalization">CONFIGURATION INJECTION (PROMPT EXTERNALIZATION)</h2> <ul> <li><strong>Architecture</strong>: Go <code>text/template</code> engine + Kubernetes <code>ConfigMap</code>.</li> <li><strong>Mount Path</strong>: <code>/etc/commander/templates/</code></li> <li><strong>Fallback</strong>: Compiles with <code>//go:embed defaults/*.tmpl</code> to ensure baseline functionality.</li> <li><strong>Purpose</strong>: Enables zero-downtime, configuration-driven prompt engineering. AI Minion personas and instructions can be modified directly via Helm <code>values.yaml</code> without requiring binary recompilation or container image rebuilds.</li> </ul>