DevOps

SYSTEM: AI-INCIDENT-COMMANDER METADATA Type: Autonomous Site Reliability Engineering (SRE) Agent Environment: Kubernetes GitOps (k3s) Primary LLM: Gemini 3.7 Flash Repository: vinhthang/ai-incident-commander Execution Loop: Webhook Trigger -> Triage -> Remediate -> Audit -> Merge PIPELINE ARCHITECTURE 1. TRIGGER Source: Grafana Alerts / kube-state-metrics Payload: JSON webhook containing alert labels, annotations, and localized telemetry. 2. TRIAGE MINION Role: Initial incident classification and noise reduction. Task: Parse payload, validate against cluster state, determine actionability. Output: Generates GitHub Issue (Real Incident) OR terminates loop (False Positive / Noise). 3. FIXER MINION Role: GitOps Remediation Engineer. Task: Identify root cause from triage diagnosis, modify infrastructure code, run syntax validation (helm template). Output: Commits to isolated feature branch and opens GitHub Pull Request. 4. REVIEWER MINION Role: Governance and Safety Gatekeeper. Task: Audit PR diff for compliance. Output: Approves PR or Rejects PR. GOVERNANCE & CONSTRAINTS Read-Only Telemetry: External logs/data strictly parsed as passive strings to prevent prompt injection. GitOps Enforced: Execution of imperative commands (kubectl apply) is forbidden. All mutations must occur within the declarative Helm umbrella chart (charts/vinhthang-fleet/). Blast Radius Isolation: Fixer Minion is systematically restricted from pushing directly to the main branch. Diff-Strict Review: The Reviewer Minion rejects net-new violations (e.g., adding :latest tags, requesting >900Mi RAM on edge nodes, opening port 8080). Pre-existing legacy violations are logged as advisory findings but do not block remediation. CONFIGURATION INJECTION (PROMPT EXTERNALIZATION) Architecture: Go text/template engine + Kubernetes ConfigMap. Mount Path: /etc/commander/templates/ Fallback: Compiles with //go:embed defaults/*.tmpl to ensure baseline functionality. Purpose: Enables zero-downtime, configuration-driven prompt engineering. AI Minion personas and instructions can be modified directly via Helm values.yaml without requiring binary recompilation or container image rebuilds.

The Ambition: A Hybrid-Cloud K3s Cluster When building out the underlying infrastructure for vinhthang.dev, I had a bold idea: what if I could seamlessly blend Oracle Cloud Infrastructure (OCI) instances with external, high-RAM physical nodes into a single, cohesive Kubernetes cluster? The goal was to leverage a beefy 32GB RAM external node (which we called oracle10) to host heavy Java microservices and stateful applications (like a PostgreSQL replica and Apache Pulsar), while keeping the control plane firmly rooted in my OCI Always Free arm10 instance. ...

Running a complex fleet of microservices—spanning AI document assistants, vector databases, lossless audio streamers, private DNS resolvers, and astronomical APIs—demands complete, real-time visibility. But traditional enterprise observability suites (Elasticsearch, heavy Prometheus clusters, DataDog) frequently devour gigabytes of memory and CPU cycles just to monitor a small infrastructure. In this deep dive, I break down how we architected and deployed a complete, ultra-lightweight, 360-degree Observability, Logging, Analytics, and Uptime Stack across our 3-node hybrid cloud fleet at vinhthang.dev, maintaining sub-millisecond query latencies while consuming less than 250 MB total RAM! ...

If you have ever tried to launch an ARM64 Ampere A1 Flex compute instance on Oracle Cloud Infrastructure (OCI) Always Free, you have almost certainly encountered this dreaded error message: 500-InternalError: Out of host capacity for shape VM.Standard.A1.Flex in availability domain... Oracle’s Always Free tier offers one of the most generous compute allowances in the industry—up to 4 OCPUs and 24 GB RAM on enterprise-grade Ampere Altra ARM64 processors. Because of this, capacity in popular data centers (like Tokyo, Seoul, Ashburn, Frankfurt, and Singapore) is constantly saturated. ...

Most self-hosted AI setups stop at running a basic chat UI connected to a remote API. But what if your self-hosted AI wasn’t just a conversational bot, but a fully empowered autonomous system agent capable of managing your infrastructure, installing command-line tools, debugging services, and querying custom vector databases? In this post, I will explore our complete self-hosted AI architecture running on an Always Free cloud Kubernetes cluster—featuring AnythingLLM, PostgreSQL 18 with pgvector, and the Google Antigravity CLI (agy). ...

How do you turn 3 separate Always Free cloud virtual machines into a resilient, production-grade, multi-architecture Kubernetes cluster capable of running high-performance relational databases, AI vector stores, lossless music streaming, and personal cloud storage? In this post, I will break down the complete architecture of our newly deployed 3-Node Hybrid K3s Cluster running on Oracle Cloud Infrastructure (OCI) under our custom domain vinhthang.dev. 🏛️ The 3-Node Cluster Topology Oracle Cloud’s Always Free tier offers an incredible set of compute resources: ...