[{"content":"Why settle for one free cloud provider when you can build an automated, resilient multi-cloud Kubernetes cluster across Oracle Cloud (OCI) and Google Cloud Platform (GCP) — completely for $0/month?\nIn this post, I walk through the end-to-end process of:\nProvisioning an Oracle Linux Server 10.1 instance on Google Cloud\u0026rsquo;s Always Free Tier (gce10). Applying kernel and daemon memory hardening to reclaim ~250 MB of RAM. Joining gce10 as a worker node into our central K3s Kubernetes cluster across a secure WireGuard mesh. Running a head-to-head performance benchmark comparing Apple Silicon M1, OCI Ampere ARM (arm10), GCP Intel Xeon (gce10), and OCI AMD EPYC (amd10/amd11). 🏛️ 1. Multi-Cloud K3s Kubernetes Fleet Architecture Our fleet previously lived entirely in OCI Tokyo. Adding gce10 in GCP Iowa (us-central1) creates a geo-distributed cross-cloud presence with a dedicated US edge outpost.\nflowchart TD subgraph OCI APAC Region [Oracle Cloud Infrastructure - Tokyo] AMD10[\u0026#34;amd10 (OCI AMD 1GB)\u0026lt;br\u0026gt;• Caddy Ingress Gateway\u0026lt;br\u0026gt;• Public IPv4: 152.70.101.162\u0026lt;br\u0026gt;• Google SSO Forward-Auth \u0026amp; WireGuard Hub\u0026#34;] AMD11[\u0026#34;amd11 (OCI AMD 1GB • K3s Worker)\u0026lt;br\u0026gt;• Pod CIDR: 10.42.1.0/24\u0026lt;br\u0026gt;• Navidrome, VietCalendar, FileBrowser\u0026#34;] ARM10[\u0026#34;arm10 (OCI Ampere ARM 10GB • K3s Master)\u0026lt;br\u0026gt;• Pod CIDR: 10.42.0.0/24\u0026lt;br\u0026gt;• PostgreSQL 18 + pgvector, VictoriaMetrics\u0026#34;] end subgraph GCP US-Central Region [Google Cloud Platform - Iowa] GCE10[\u0026#34;gce10 (GCP e2-micro 1GB • K3s Worker)\u0026lt;br\u0026gt;• Pod CIDR: 10.42.2.0/24 (Flannel over WireGuard)\u0026lt;br\u0026gt;• Public IPv4: 136.111.37.17\u0026lt;br\u0026gt;• Vector Log Shipper \u0026amp; US Edge Bridge\u0026#34;] end AMD10 \u0026lt;--\u0026gt;|Internal VCN| ARM10 AMD10 \u0026lt;--\u0026gt;|Internal VCN| AMD11 GCE10 \u0026lt;--\u0026gt;|WireGuard Mesh wg0 (10.10.0.4 \u0026lt;-\u0026gt; 10.10.0.1)| AMD10 GCE10 \u0026lt;--\u0026gt;|Cross-Cloud Flannel vxlan / Pod Routing| ARM10 🎁 2. Google Cloud Always Free Tier: Rules \u0026amp; Gotchas Google Cloud provides one of the most generous persistent free tiers, but you must configure your VM strictly within these boundaries to avoid unexpected billing:\nParameter Always Free Limit Our Selected Config Gotcha to Avoid Machine Type 1 non-preemptible e2-micro / month e2-micro (2 vCPU, 1 GB RAM) Do NOT pick e2-small or e2-medium. Region us-central1, us-east1, or us-west1 us-central1-a (Iowa) Non-US regions (Singapore, Tokyo) are NOT free! Boot Disk Up to 30 GB Standard Persistent Disk 30 GB pd-standard Do NOT choose pd-balanced or pd-ssd. Outbound Egress 1 GB/month to all destinations Standard egress Bulk media streaming will exceed 1 GB. Provisioning Command via gcloud CLI gcloud compute instances create gce-free-vm \\ --project=vietcalendar \\ --zone=us-central1-a \\ --machine-type=e2-micro \\ --image-family=oracle-linux-10 \\ --image-project=oracle-linux-cloud \\ --boot-disk-size=30GB \\ --boot-disk-type=pd-standard \\ --boot-disk-auto-delete \\ --metadata-from-file=ssh-keys=/path/to/ssh-keys.txt \\ --tags=http-server,https-server 🧠 3. Memory Optimization Playbook on a 1 GB Enterprise VM Oracle Linux 10 boots with the Unbreakable Enterprise Kernel (UEK 6.12) and standard enterprise daemons. Out-of-the-box, it consumed ~600 MiB of RAM, leaving only ~350 MiB for user workloads.\nHere is how we reclaimed memory and stabilized the baseline before starting K3s:\nStep 1: Disable kdump \u0026amp; Reclaim Kernel crashkernel RAM sudo systemctl disable --now kdump sudo grubby --update-kernel=ALL --remove-args=\u0026#34;crashkernel crash_kexec_post_notifiers\u0026#34; Step 2: Decommission Heavy \u0026amp; Redundant Background Daemons Google Guest Agent Suite (~81 MiB freed): Masked google-guest-agent-manager, google-guest-compat-manager, and core_plugin. Redundant Linux Daemons (~60 MiB freed): Masked firewalld, tuned, google-osconfig-agent, rsyslog, rngd, dtprobed, auditd, and rpcbind. Step 3: Setup 2.0 GB Swapfile with Low Swappiness sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile echo \u0026#39;/swapfile none swap sw 0 0\u0026#39; | sudo tee -a /etc/fstab echo \u0026#39;vm.swappiness=10\u0026#39; | sudo tee /etc/sysctl.d/99-swap.conf sudo sysctl -p /etc/sysctl.d/99-swap.conf ☸️ 4. Joining K3s Kubernetes Over a WireGuard Mesh With gce10 connected to amd10 over WireGuard (10.10.0.4 $\\leftrightarrow$ 10.10.0.1), we joined gce10 to the master on arm10 (10.0.0.216):\ncurl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_SELINUX_RPM=true \\ INSTALL_K3S_VERSION=\u0026#39;v1.36.3+k3s1\u0026#39; \\ K3S_URL=\u0026#39;https://10.0.0.216:6443\u0026#39; \\ K3S_TOKEN=\u0026#39;\u0026lt;MASTER_NODE_TOKEN\u0026gt;\u0026#39; \\ sh -s - agent \\ --node-name gce10 \\ --node-ip 10.10.0.4 \\ --node-external-ip 136.111.37.17 \\ --flannel-iface wg0 Verification on arm10 Master NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION amd11 Ready worker 3d6h v1.36.3+k3s1 10.0.0.10 \u0026lt;none\u0026gt; Oracle Linux Server 10.2 6.12.0-204.92.4.4.3.el10uek.x86_64 arm10 Ready control-plane 3d6h v1.36.3+k3s1 10.0.0.216 \u0026lt;none\u0026gt; Oracle Linux Server 10.2 6.12.0-204.92.4.4.3.el10uek.aarch64 gce10 Ready worker 2m v1.36.3+k3s1 10.10.0.4 136.111.37.17 Oracle Linux Server 10.1 6.12.0-202.76.4.1.el10uek.x86_64 The cluster\u0026rsquo;s vector daemonset instantly scheduled onto gce10 (Pod IP: 10.42.2.2), immediately harvesting logs and streaming them back to VictoriaLogs on arm10!\n⚡ 5. The 5-Node Benchmark Showdown We ran an identical, standardized benchmark suite across 5 environments:\n🍎 Apple MacBook Air (M1, 8 Cores, 16 GB Unified Memory) 🚀 arm10 (OCI Ampere Altra ARM64, 2 OCPUs, 10.9 GB RAM) 🌐 gce10 (GCP Intel Xeon x86_64, 2 vCPUs, 1 GB RAM) 🛡️ amd10 (OCI AMD EPYC 7551 x86_64, 2 vCPUs, 1 GB RAM) 🛡️ amd11 (OCI AMD EPYC 7551 x86_64, 2 vCPUs, 1 GB RAM) A. CPU Compute (Prime Calculation — Lower is Faster) Node Processor Single-Core Time 2-Core Multi-Thread Relative Speed 🍎 Mac (M1) Apple M1 (ARM64) 0.028 s 🏆 0.192 s 6.4x faster than AMD arm10 Ampere Altra (ARM64) 0.049 s 🥈 0.211 s 3.6x faster than AMD gce10 Intel Xeon (x86_64) 0.066 s 🥉 0.194 s 🥇 3.0x faster than AMD amd10 AMD EPYC 7551 (x86_64) 0.178 s 0.587 s Baseline amd11 AMD EPYC 7551 (x86_64) 0.201 s 0.600 s Baseline B. Direct Disk I/O (Bypassing RAM Cache — Higher is Better) Node Direct Write (128 MB) Direct Read (128 MB) Storage Medium 🍎 Mac (M1) 464.8 MB/s 🏆 13,724 MB/s 🏆 Apple Internal PCIe NVMe SSD gce10 87.2 MB/s 🥇 140.6 MB/s 🥇 GCP Standard Persistent Disk (30 GB) amd10 47.5 MB/s 🥈 73.5 MB/s 🥈 OCI Block Storage (50 GB) amd11 27.7 MB/s 49.8 MB/s OCI Block Storage (50 GB) arm10 17.7 MB/s 10.0 MB/s OCI Block Storage (Active cluster I/O) 🎯 6. Fleet Specialization Matrix arm10 (OCI ARM 10 GB): The Core Engine. Houses our PostgreSQL 18 database with pgvector, Spring Boot microservices, VictoriaMetrics TSDB, and K3s control plane. gce10 (GCP Intel 1 GB): The US Outpost \u0026amp; Worker. High single-core clock speed, fast 140 MB/s storage I/O, and K3s worker node in North America. amd10 \u0026amp; amd11 (OCI AMD 1 GB): The Gatekeepers \u0026amp; Shields. Caddy Ingress \u0026amp; TLS Termination: Caddy uses \u0026lt; 30 MB RAM and \u0026lt; 2% CPU to handle all public domain routing. 2 Free Dedicated Public IPv4s: OCI includes static public IPs for free. Zero-Trust Bastion \u0026amp; WireGuard Mesh: Encrypted overlay networking between private subnets and cross-cloud nodes. ","permalink":"https://vinhthang.dev/posts/gcp-always-free-oracle-linux-10-cross-cloud/","summary":"\u003cp\u003eWhy settle for one free cloud provider when you can build an automated, resilient \u003cstrong\u003emulti-cloud Kubernetes cluster across Oracle Cloud (OCI) and Google Cloud Platform (GCP)\u003c/strong\u003e — completely for \u003cstrong\u003e$0/month\u003c/strong\u003e?\u003c/p\u003e\n\u003cp\u003eIn this post, I walk through the end-to-end process of:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eProvisioning an \u003cstrong\u003eOracle Linux Server 10.1\u003c/strong\u003e instance on Google Cloud\u0026rsquo;s \u003cstrong\u003eAlways Free Tier (\u003ccode\u003egce10\u003c/code\u003e)\u003c/strong\u003e.\u003c/li\u003e\n\u003cli\u003eApplying kernel and daemon memory hardening to reclaim ~250 MB of RAM.\u003c/li\u003e\n\u003cli\u003eJoining \u003ccode\u003egce10\u003c/code\u003e as a worker node into our central \u003cstrong\u003eK3s Kubernetes cluster\u003c/strong\u003e across a secure \u003cstrong\u003eWireGuard mesh\u003c/strong\u003e.\u003c/li\u003e\n\u003cli\u003eRunning a head-to-head performance benchmark comparing \u003cstrong\u003eApple Silicon M1\u003c/strong\u003e, \u003cstrong\u003eOCI Ampere ARM (\u003ccode\u003earm10\u003c/code\u003e)\u003c/strong\u003e, \u003cstrong\u003eGCP Intel Xeon (\u003ccode\u003egce10\u003c/code\u003e)\u003c/strong\u003e, and \u003cstrong\u003eOCI AMD EPYC (\u003ccode\u003eamd10\u003c/code\u003e/\u003ccode\u003eamd11\u003c/code\u003e)\u003c/strong\u003e.\u003c/li\u003e\n\u003c/ol\u003e\n\u003chr\u003e\n\u003ch2 id=\"-1-multi-cloud-k3s-kubernetes-fleet-architecture\"\u003e🏛️ 1. Multi-Cloud K3s Kubernetes Fleet Architecture\u003c/h2\u003e\n\u003cp\u003eOur fleet previously lived entirely in OCI Tokyo. Adding \u003cstrong\u003e\u003ccode\u003egce10\u003c/code\u003e\u003c/strong\u003e in GCP Iowa (\u003ccode\u003eus-central1\u003c/code\u003e) creates a geo-distributed cross-cloud presence with a dedicated US edge outpost.\u003c/p\u003e","title":"🌐 Expanding to Multi-Cloud: Provisioning Oracle Linux 10 on Google Cloud's $0 Always Free Tier \u0026 Joining K3s"},{"content":"What can you actually run on a $0/month Always Free cloud server?\nIn this post, I document the end-to-end journey of turning an Oracle Cloud 1 GB AMD Micro instance (amd10) in Tokyo into a multi-service personal cloud hub — complete with Dual-Stack DNS ad-blocking, Google Single Sign-On (SSO), lossless Hi-Res music streaming, a Rust-powered REST \u0026amp; AI MCP API, and a Hugo static blog.\n🏗️ Architecture \u0026amp; Network Blueprint graph TD Internet[\u0026#34;🌐 Internet / Home Network / Mobile\u0026#34;] --\u0026gt; Caddy[\u0026#34;🔒 Caddy (Auto Let\u0026#39;s Encrypt SSL)\u0026#34;] subgraph OCI Cloud Instance (amd10 - Tokyo) Caddy --\u0026gt;|/| VietCal[\u0026#34;📅 VietCalendar REST \u0026amp; MCP (Rust :8080)\u0026#34;] Caddy --\u0026gt;|files.*| Oauth[\u0026#34;🔑 OAuth2-Proxy (Google SSO :4180)\u0026#34;] Oauth --\u0026gt; FileBrowser[\u0026#34;📁 FileBrowser Cloud Drive (:8082)\u0026#34;] Caddy --\u0026gt;|music.*| Navidrome[\u0026#34;🎵 Navidrome Lossless FLAC (:4533)\u0026#34;] Caddy --\u0026gt;|blog.*| Hugo[\u0026#34;📝 Hugo Static Site (:443)\u0026#34;] Caddy --\u0026gt;|adguard.*| AdGuardWeb[\u0026#34;🛡️ AdGuard Dashboard (:3000)\u0026#34;] Router[\u0026#34;🏠 Home Router (Fiber IPv4 \u0026amp; IPv6)\u0026#34;] --\u0026gt; AdGuardDNS[\u0026#34;🛡️ AdGuard DNS Server (:53 \u0026amp; :853 DoT)\u0026#34;] end ⚡ 1. Reclaiming Hardware RAM (The crashkernel Discovery) On a standard 1 GB VM, Linux kdump can silently reserve up to 448 MB RAM (nearly half the server\u0026rsquo;s memory!).\nBy disabling crashkernel=no in GRUB:\nReclaimed physical RAM from 512 MB → 946 MB. Added a 2.5 GB swap file with zram/swappiness tuning. Result: Plenty of headroom to run 6 concurrent services simultaneously. 🛡️ 2. Whole-Home Dual-Stack Ad-Blocking \u0026amp; Private DNS We provisioned a dual-stack network with reserved static addresses:\nPublic IPv4: 152.70.xxx.xxx (Always Free Reserved Static IP) Public IPv6: 2603:c021:8022:100::xxxx (Always Free /56 Prefix) Android / iOS Private DNS (DoT): \u0026lt;your-domain\u0026gt;.duckdns.org on Port 853 with automatic Let\u0026rsquo;s Encrypt TLS. Router DNS: Configured on our home fiber router, blocking ads and tracking telemetry across Smart TVs, iPhones, laptops, and IoT devices. 📅 3. VietCalendar: High-Performance Rust \u0026amp; AI MCP Server Running on native Rust (Axum framework):\nComputes accurate Vietnamese Solar-to-Lunar conversions and public holiday schedules. Exposes an interactive Swagger OpenAPI UI at /swagger-ui/. Exposes a Universal Model Context Protocol (MCP) endpoint at /mcp/sse for AI assistants like Antigravity and Claude. RAM footprint: An astonishing \u0026lt; 1 MB RAM! 🎵 4. Navidrome: Bit-Perfect Lossless FLAC Streaming Deployed Navidrome in Go to stream Hi-Res 24-bit / 96kHz FLAC and ALAC audio. Full Subsonic API compatibility with audiophile mobile apps: Android: Symfonium (bit-perfect USB DAC output). iOS: Amperfy and play:Sub. RAM footprint: ~48 MB RAM. 📁 5. FileBrowser with Google Single Sign-On (SSO) Self-hosted cloud drive on the 38 GB free NVMe disk. Secured by OAuth2-Proxy: Whitelisted strictly to personal Google accounts. No passwords required — 1 click \u0026ldquo;Sign In with Google\u0026rdquo; logs you in directly. Seamless Synergy: Uploading audio files into the Music/ folder in FileBrowser automatically syncs them into Navidrome for streaming! Blog Synergy: Markdown posts created in the Blog/ folder auto-publish to this website! 📝 6. Hugo + PaperMod: Sub-Millisecond Static Tech Blog Static site generator built with Hugo and the PaperMod theme. Served directly by Caddy with gzip/zstd compression. 0 MB RAM runtime overhead and instant sub-second auto-rebuilding whenever a Markdown post is saved. 📊 Live Server Resource Breakdown (amd10) All 6 services running concurrently on 1 single core and 1 GB RAM:\nService Port Memory Usage Status 📁 FileBrowser 8082 18.1 MB 🟢 Active 🔑 OAuth2-Proxy 4180 22.8 MB 🟢 Active 🎵 Navidrome 4533 53.0 MB 🟢 Active 🛡️ AdGuard Home 53 / 853 104.7 MB 🟢 Active 📅 VietCalendar 8080 0.8 MB 🟢 Active 📝 Hugo Auto-Rebuilder systemd 26.8 MB 🟢 Active 🔒 Caddy Web Server 80 / 443 ~25.0 MB 🟢 Active TOTAL FREE MEMORY — ~320 MB Free Physical RAM + 2.2 GB Swap 🟢 Healthy 💻 Infrastructure as Code \u0026amp; Open Source The complete automated provisioning runbook, security lists, and Terraform configurations are open-sourced on GitHub:\n👉 https://github.com/vinhthang/oci\nWritten on and served from amd10 in Tokyo.\n","permalink":"https://vinhthang.dev/posts/building-my-always-free-cloud/","summary":"\u003cp\u003eWhat can you actually run on a \u003cstrong\u003e$0/month Always Free cloud server\u003c/strong\u003e?\u003c/p\u003e\n\u003cp\u003eIn this post, I document the end-to-end journey of turning an Oracle Cloud \u003cstrong\u003e1 GB AMD Micro instance (\u003ccode\u003eamd10\u003c/code\u003e)\u003c/strong\u003e in Tokyo into a multi-service personal cloud hub — complete with \u003cstrong\u003eDual-Stack DNS ad-blocking\u003c/strong\u003e, \u003cstrong\u003eGoogle Single Sign-On (SSO)\u003c/strong\u003e, \u003cstrong\u003elossless Hi-Res music streaming\u003c/strong\u003e, a \u003cstrong\u003eRust-powered REST \u0026amp; AI MCP API\u003c/strong\u003e, and a \u003cstrong\u003eHugo static blog\u003c/strong\u003e.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"-architecture--network-blueprint\"\u003e🏗️ Architecture \u0026amp; Network Blueprint\u003c/h2\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode class=\"language-mermaid\" data-lang=\"mermaid\"\u003egraph TD\n    Internet[\u0026#34;🌐 Internet / Home Network / Mobile\u0026#34;] --\u0026gt; Caddy[\u0026#34;🔒 Caddy (Auto Let\u0026#39;s Encrypt SSL)\u0026#34;]\n    \n    subgraph OCI Cloud Instance (amd10 - Tokyo)\n        Caddy --\u0026gt;|/| VietCal[\u0026#34;📅 VietCalendar REST \u0026amp; MCP (Rust :8080)\u0026#34;]\n        Caddy --\u0026gt;|files.*| Oauth[\u0026#34;🔑 OAuth2-Proxy (Google SSO :4180)\u0026#34;]\n        Oauth --\u0026gt; FileBrowser[\u0026#34;📁 FileBrowser Cloud Drive (:8082)\u0026#34;]\n        Caddy --\u0026gt;|music.*| Navidrome[\u0026#34;🎵 Navidrome Lossless FLAC (:4533)\u0026#34;]\n        Caddy --\u0026gt;|blog.*| Hugo[\u0026#34;📝 Hugo Static Site (:443)\u0026#34;]\n        Caddy --\u0026gt;|adguard.*| AdGuardWeb[\u0026#34;🛡️ AdGuard Dashboard (:3000)\u0026#34;]\n        \n        Router[\u0026#34;🏠 Home Router (Fiber IPv4 \u0026amp; IPv6)\u0026#34;] --\u0026gt; AdGuardDNS[\u0026#34;🛡️ AdGuard DNS Server (:53 \u0026amp; :853 DoT)\u0026#34;]\n    end\n\u003c/code\u003e\u003c/pre\u003e\u003chr\u003e\n\u003ch2 id=\"-1-reclaiming-hardware-ram-the-crashkernel-discovery\"\u003e⚡ 1. Reclaiming Hardware RAM (The \u003ccode\u003ecrashkernel\u003c/code\u003e Discovery)\u003c/h2\u003e\n\u003cp\u003eOn a standard 1 GB VM, Linux kdump can silently reserve up to \u003cstrong\u003e448 MB RAM\u003c/strong\u003e (nearly half the server\u0026rsquo;s memory!).\u003c/p\u003e","title":"🛠️ Building a Production-Grade Home Cloud on Oracle's $0 Always Free Tier"},{"content":"Running a complex fleet of microservices—spanning AI document assistants, vector databases, lossless audio streamers, private DNS resolvers, and astronomical APIs—demands complete, real-time visibility. But traditional enterprise observability suites (Elasticsearch, heavy Prometheus clusters, DataDog) frequently devour gigabytes of memory and CPU cycles just to monitor a small infrastructure.\nIn this deep dive, I break down how we architected and deployed a complete, ultra-lightweight, 360-degree Observability, Logging, Analytics, and Uptime Stack across our 3-node hybrid cloud fleet at vinhthang.dev, maintaining sub-millisecond query latencies while consuming less than 250 MB total RAM!\n🏛️ The 5 Pillars of Our Modern Telemetry Architecture Instead of deploying bloated monolithic logging tools, we engineered a modular, decoupled architecture where each layer excels at its specific domain:\n[ 🌐 Public Traffic \u0026amp; Visitors ] │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 1. Edge Gateway (amd10) │ │ • Caddy TLS Termination \u0026amp; Reverse Proxy │ │ • Central Google OAuth2 Single Sign-On (auth.vinhthang.dev) │ │ • Privacy-Friendly Blog Analytics Snippet (analytics.vinhthang.dev) │ └──────────────────────────────────────────────┬──────────────────────────────────────────────────┘ │ (Private Oracle VCN 10.0.0.0/16) ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 2. Kubernetes Cluster (arm10 \u0026amp; amd11) │ │ │ │ 📊 TSDB METRICS LAYER 🪵 LOGGING LAYER 🔭 DISTRIBUTED TRACING LAYER │ │ • VictoriaMetrics (Port 8428) • VictoriaLogs (Port 9428) • OpenTelemetry Collector │ │ • PromQL Compatible (\u0026lt; 35MB) • Promtail Multi-Node Daemon • OTLP gRPC (:4317) \u0026amp; HTTP │ │ • 30-Day Auto Retention • 15x Log Compression (\u0026lt; 40MB) • Application Tracing Spans │ │ │ │ 🟢 WATCHDOG HEALTH LAYER 📈 VISITOR ANALYTICS LAYER 🐘 DATA PERSISTENCE LAYER │ │ • Uptime Kuma 2.5.3 (Port 30002) • Umami Analytics (Port 30003) • PostgreSQL 18.6 + pgvector │ │ • 9 Cloud Monitors + Status Page • Relational Web Visitor DB • AI Vectors \u0026amp; Memos Storage │ └──────────────────────────────────────────────┬──────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 3. Unified Visualization Dashboard │ │ 📈 Grafana 11 Command Center (grafana.vinhthang.dev) │ │ • Google SSO Auto-Login • Multi-Engine DataSources (PromQL, LogsQL, SQL) │ └─────────────────────────────────────────────────────────────────────────────────────────────────┘ 📊 1. Metrics Layer: VictoriaMetrics (High-Performance PromQL TSDB) Traditional Prometheus TSDBs can easily consume 500 MB to 1 GB of memory for indexing timeseries metrics. To keep our footprint microscopic, we selected VictoriaMetrics:\nRAM Footprint: \u0026lt; 35 MB RAM (10x lighter than Prometheus). Native PromQL: 100% drop-in compatible with standard Prometheus formulas and Grafana community dashboards. Storage Compression: Gorilla delta-of-delta encoding, achieving less than 1.5 bytes per metric datapoint with automated 30-day data retention. Automatic Scraping: VictoriaMetrics scrapes Uptime Kuma (:3001/metrics) and OpenTelemetry Collector every 30 seconds. 🪵 2. Logging Layer: VictoriaLogs \u0026amp; Multi-Node Promtail For container and system log aggregation, we bypassed the heavy Elasticsearch/Loki memory footprint and deployed VictoriaLogs:\nUltra-Lean Engine: VictoriaLogs uses only ~40 MB RAM on arm10. 15x Log Compression: Compresses raw JSON/CRI container logs by over 90%, storing millions of log lines in a few megabytes of disk. Multi-Node Promtail DaemonSet: Deployed across all nodes (arm10 and amd11) to automatically harvest and stream container logs from /var/log/pods/*/*/*.log. Intuitive LogsQL: Querying logs is blazing fast without index bloat: # Search all logs across pods * # Search specific microservice errors app:anythingllm AND error # Aggregate error rates error | stats count() by (container) 🟢 3. Watchdog \u0026amp; Status Layer: Uptime Kuma v2.5.3 For synthetic health monitoring and external availability testing, we upgraded to Uptime Kuma v2.5.3:\nDecoupled Architecture: Uses embedded SQLite with WAL mode to ensure the health monitor remains 100% operational even if the central database restarts. Continuous Probing: Probes all 9 microservices across HTTPS, TLS handshakes, HTTP response codes, and SSL certificate expiration countdowns. Public Status Page: Hosted at https://status.vinhthang.dev, secured via Central Google Single Sign-On for administrative control. 📈 4. Web \u0026amp; Visitor Analytics: Umami + PostgreSQL 18.6 Instead of injecting privacy-invasive third-party scripts (like Google Analytics), we self-host Umami Analytics:\nCookieless \u0026amp; GDPR Compliant: 100% anonymous visitor tracking with zero user fingerprinting or personal data collection. PostgreSQL 18 Integration: All pageviews, referrers, devices, and geographic insights are stored directly in PostgreSQL 18.6 on arm10. Accessible via: https://analytics.vinhthang.dev. 📈 5. Command Center: Grafana 11 Single-Pane-of-Glass To bring everything together, we deployed Grafana 11 at https://grafana.vinhthang.dev:\nSingle Sign-On Auth Proxy: Automatically logs you in as Admin via Google SSO (thanghv@gmail.com) with zero secondary password friction. Multi-Engine Telemetry Matrix: VictoriaMetrics (PromQL): Real-time service status, latency wave charts, SSL expiration gauges. VictoriaLogs (LogsQL): Live streaming container logs with full-text search. PostgreSQL 18 (SQL): Relational notes from Memos, tracked traffic events from Umami, and pgvector AI index health. ⚡ Real-World Benchmarking: 100 Concurrent Requests To verify that our monitoring and stateless microservices can handle burst traffic, we subjected our VietCalendar Rust API (api.vinhthang.dev) to a 100-request concurrent load test through Cloudflare and Caddy:\nTotal Requests: 100 / 100 Success Rate: 100.00% 200 OK (0 errors) Average Latency: 801 ms (including end-to-end public TLS negotiation) Throughput: 10.3 requests/sec Memory Spike: 0 MB (Rust astronomical calculations executed purely in CPU registers). 📊 Complete Fleet Memory Footprint Here is the actual memory allocation across the entire cloud fleet after deploying the full observability stack:\nNode Total RAM Used RAM Free Headroom Available Active Workloads arm10 (ARM64 A1) 10.0 GiB 2.8 GiB (28%) 🟢 7.9 GiB (72% FREE) K3s Master, PG 18 + pgvector, AnythingLLM, VictoriaMetrics, VictoriaLogs, Grafana 11, OTel, Umami, Uptime Kuma. amd10 (Edge Ingress) 946 MiB 470 MiB (49%) 🟢 476 MiB (51% FREE) Caddy Edge Gateway, AdGuard Home DoT (853), Hugo Static Blog. amd11 (Worker) 945 MiB 527 MiB (55%) 🟢 418 MiB (45% FREE) K3s Agent, Navidrome FLAC, FileBrowser Storage, VietCalendar (Rust). 🚀 Key Takeaways You Don\u0026rsquo;t Need Massive Cloud Budgets for Enterprise Observability: By combining VictoriaMetrics, VictoriaLogs, and Grafana, you achieve multi-node metrics, full-text logging, and distributed tracing in less than 250 MB RAM. Decoupled Architecture Wins: Separating stateless APIs (Rust), synthetic watchdogs (Uptime Kuma), relational stores (PostgreSQL 18), and time-series TSDBs prevents cascading outages. Single Sign-On at the Edge: Wrapping your monitoring dashboards in a central OAuth2 gateway guarantees security with zero login fatigue. All Kubernetes manifests, Promtail configs, and Grafana dashboard definitions are open-source and codified in our Git repository! 🌟🥂\n","permalink":"https://vinhthang.dev/posts/complete-observability-stack-metrics-logs-traces-2026/","summary":"\u003cp\u003eRunning a complex fleet of microservices—spanning AI document assistants, vector databases, lossless audio streamers, private DNS resolvers, and astronomical APIs—demands complete, real-time visibility. But traditional enterprise observability suites (Elasticsearch, heavy Prometheus clusters, DataDog) frequently devour gigabytes of memory and CPU cycles just to monitor a small infrastructure.\u003c/p\u003e\n\u003cp\u003eIn this deep dive, I break down how we architected and deployed a \u003cstrong\u003ecomplete, ultra-lightweight, 360-degree Observability, Logging, Analytics, and Uptime Stack\u003c/strong\u003e across our 3-node hybrid cloud fleet at \u003ca href=\"https://vinhthang.dev\"\u003e\u003cstrong\u003e\u003ccode\u003evinhthang.dev\u003c/code\u003e\u003c/strong\u003e\u003c/a\u003e, maintaining sub-millisecond query latencies while consuming \u003cstrong\u003eless than 250 MB total RAM\u003c/strong\u003e!\u003c/p\u003e","title":"Building an Ultra-Lightweight Cloud-Native Observability \u0026 Analytics Stack: Metrics, Logs, Traces \u0026 Status on OCI"},{"content":"One of the most frustrating bottlenecks in self-hosting is uploading large files to overseas cloud servers.\nWhether you are uploading a 45 GB lossless FLAC music collection to Navidrome, backing up 4K video files to FileBrowser, or transferring gigabytes of PDF datasets to an AI vector store, standard public internet routing often degrades performance to an agonizing crawl.\nIn this post, I will explain why international cross-border uploads stall, and how we used Cloudflare WARP and Anycast Edge Ingestion to boost our file upload throughput by 5x to 10x directly into our cloud fleet at vinhthang.dev.\n🐌 The Bottleneck: Why Public Internet Uploads Are Slow When you upload a large file from your local workstation to a cloud server located in another country (such as an Oracle Cloud data center in Tokyo or US-Ashburn), your traffic travels through the public internet:\n[ 💻 Local Workstation ] │ (High packet loss \u0026amp; ISP throttling) ▼ [ 🌐 Congested Public Internet ] ── (15+ Slow BGP Transit Hops) ──▶ [ ☁️ Cloud Host ] Why This Destroys Upload Speed: Submarine Cable Congestion \u0026amp; Packet Loss: Even a 0.5% packet loss causes TCP congestion control (like CUBIC or Reno) to cut upload throughput by over 70%. Sub-Optimal BGP Routing: ISPs frequently route packets through cheaper, circuitous international paths rather than the shortest geographical distance. High Round-Trip Time (RTT): High latency between your client and origin server causes TCP window scaling and HTTP multipart chunk acknowledgments to stall. 🚀 The Solution: Cloudflare WARP Anycast Acceleration Cloudflare WARP transforms your client connection by routing traffic into Cloudflare\u0026rsquo;s ultra-low-latency private fiber backbone:\n[ 💻 Local Workstation ] │ │ (WireGuard / UDP tunnel in \u0026lt; 5ms) ▼ [ ⚡ Nearest Cloudflare Edge POP (Hanoi / HCMC / Singapore) ] │ │ (Private High-Speed Optical Backbone / Zero Packet Loss) ▼ [ ☁️ Cloudflare Tokyo Edge ] ── (Direct Local Peering) ──▶ [ 🛡️ Origin Server: amd10 ] ⚡ How Cloudflare WARP Turbocharges File Ingestion 1. Ultra-Low First-Mile Latency (WireGuard Engine) WARP uses a heavily optimized Rust implementation of the WireGuard protocol (BoringTun). Your computer establishes a persistent UDP tunnel to the nearest Cloudflare Edge Point of Presence (POP) in less than 5 milliseconds. TCP connection handshakes and SSL/TLS negotiations occur locally at the edge rather than across the ocean. 2. Private Global Backbone (Bypassing the Public Internet) Once your data reaches the local Cloudflare POP, it exits the public internet entirely. Your file chunks travel across Cloudflare\u0026rsquo;s private 300+ Tbps global optical network, which features dedicated routing, zero ISP throttling, and near-zero packet loss. 3. Optimized Edge Ingress Configuration in Caddy To ensure our origin server on amd10 ingested high-speed streams without buffer stalls, we tuned our Caddy Edge Gateway with non-blocking streaming:\n# FileBrowser Personal Cloud Storage on vinhthang.dev files.vinhthang.dev { # Allow up to 20 GB single file uploads request_body { max_size 20GB } # Disable response buffering for immediate real-time feedback reverse_proxy 10.0.0.10:8082 { flush_interval -1 } } 📊 Real-World Performance Benchmark We tested uploading a 2.5 GB lossless audio archive to our self-hosted FileBrowser instance before and after enabling Cloudflare WARP:\nConnection Mode Average Upload Speed Transfer Time Packet Loss ❌ Standard Public ISP Routing 4.2 MB/s ~10m 15s ~0.8% ⚡ Cloudflare WARP Client Enabled 48.6 MB/s ~52 seconds! 0.0% 🚀 Result: A 10x+ real-world speed improvement and instant saturation of our local broadband uplink!\n🛠️ How to Enable Cloudflare WARP for Your Workstation On macOS / Linux: # 1. Install Cloudflare WARP CLI brew install --cask cloudflare-warp # macOS # or: sudo apt install cloudflare-warp # Ubuntu/Debian # 2. Register and Connect warp-cli registration new warp-cli connect # 3. Verify Connection Status warp-cli status Once connected, all outbound traffic to your custom domain (https://files.vinhthang.dev or https://ai.vinhthang.dev) automatically flows through the nearest Cloudflare Anycast edge!\n🎯 Key Takeaways Don\u0026rsquo;t let public ISP routing bottleneck your cloud servers: Moving traffic onto an Anycast edge network like Cloudflare WARP eliminates international packet loss and latency spikes. Combine WARP with HTTP/3 \u0026amp; non-blocking reverse proxies: Ensuring your origin gateway (Caddy / Nginx) disables upload buffering prevents local memory exhaustion on small cloud VMs. Enjoy seamless high-speed backups: Uploading gigabytes of FLAC music, Docker images, and AI documents is now as fast as copying to a local NAS! Check out our full cloud fleet architecture at vinhthang.dev!\n","permalink":"https://vinhthang.dev/posts/supercharging-uploads-cloudflare-warp/","summary":"\u003cp\u003eOne of the most frustrating bottlenecks in self-hosting is \u003cstrong\u003euploading large files to overseas cloud servers\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eWhether you are uploading a 45 GB lossless FLAC music collection to Navidrome, backing up 4K video files to FileBrowser, or transferring gigabytes of PDF datasets to an AI vector store, standard public internet routing often degrades performance to an agonizing crawl.\u003c/p\u003e\n\u003cp\u003eIn this post, I will explain why international cross-border uploads stall, and how we used \u003cstrong\u003eCloudflare WARP\u003c/strong\u003e and \u003cstrong\u003eAnycast Edge Ingestion\u003c/strong\u003e to boost our file upload throughput by \u003cstrong\u003e5x to 10x\u003c/strong\u003e directly into our cloud fleet at \u003ca href=\"https://vinhthang.dev\"\u003e\u003cstrong\u003e\u003ccode\u003evinhthang.dev\u003c/code\u003e\u003c/strong\u003e\u003c/a\u003e.\u003c/p\u003e","title":"Supercharging Self-Hosted Cloud Uploads: How Cloudflare WARP Turbocharges File Ingestion"},{"content":"If you have ever tried to launch an ARM64 Ampere A1 Flex compute instance on Oracle Cloud Infrastructure (OCI) Always Free, you have almost certainly encountered this dreaded error message:\n500-InternalError: Out of host capacity for shape VM.Standard.A1.Flex in availability domain...\nOracle\u0026rsquo;s Always Free tier offers one of the most generous compute allowances in the industry—up to 4 OCPUs and 24 GB RAM on enterprise-grade Ampere Altra ARM64 processors. Because of this, capacity in popular data centers (like Tokyo, Seoul, Ashburn, Frankfurt, and Singapore) is constantly saturated.\nClicking \u0026ldquo;Create\u0026rdquo; manually in the web console is an exercise in futility. In this post, I will share the exact automated strategy and Python script we used to reliably capture our arm10 powerhouse instance.\n🎯 The Strategy: Why Manual Creation Fails Cloud capacity fluctuates dynamically. When other users terminate test instances, decommission pods, or resize workloads, capacity becomes available for mere seconds before being claimed.\nTo catch these capacity windows, you need an automated process that:\nPolls OCI Compute APIs continuously in a loop. Handles rate limits and API backoff gracefully. Pre-provisions all networking, cloud-init scripts, and SSH keys so the instance immediately boots into a ready state the instant capacity appears. 🛠️ The Automated Python Provisioner Script Here is the clean, production-ready Python script using the official oci Python SDK:\n#!/usr/bin/env python3 \u0026#34;\u0026#34;\u0026#34; Oracle Cloud Infrastructure (OCI) ARM64 Ampere A1 Automated Provisioner Continuously attempts to create an A1.Flex compute instance until capacity is secured. \u0026#34;\u0026#34;\u0026#34; import time import sys import oci # ============================================================================== # Configuration Parameters (Replace with your compartment and subnet OCIDs) # ============================================================================== COMPARTMENT_ID = \u0026#34;ocid1.compartment.oc1..example_compartment_ocid\u0026#34; SUBNET_ID = \u0026#34;ocid1.subnet.oc1..example_subnet_ocid\u0026#34; IMAGE_ID = \u0026#34;ocid1.image.oc1..example_oracle_linux_arm64_image_ocid\u0026#34; SSH_PUBLIC_KEY = \u0026#34;ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... user@workstation\u0026#34; INSTANCE_NAME = \u0026#34;arm10\u0026#34; OCPUS = 2 # 2 OCPUs (out of 4 free allowed) MEMORY_IN_GBS = 12 # 12 GB RAM (out of 24 GB free allowed) BOOT_VOLUME_GB = 50 # Boot volume size in GB def create_instance_client(): # Load configuration from ~/.oci/config config = oci.config.from_file() return oci.core.ComputeClient(config), config def launch_arm_instance(): compute_client, config = create_instance_client() # 1. Discover Availability Domain identity_client = oci.identity.IdentityClient(config) ads = identity_client.list_availability_domains(COMPARTMENT_ID).data target_ad = ads[0].name print(f\u0026#34;🎯 Target Availability Domain: {target_ad}\u0026#34;) # 2. Build Instance Launch Details launch_details = oci.core.models.LaunchInstanceDetails( compartment_id=COMPARTMENT_ID, availability_domain=target_ad, display_name=INSTANCE_NAME, shape=\u0026#34;VM.Standard.A1.Flex\u0026#34;, shape_config=oci.core.models.LaunchInstanceShapeConfigDetails( ocpus=OCPUS, memory_in_gbs=MEMORY_IN_GBS ), source_details=oci.core.models.InstanceSourceViaImageDetails( image_id=IMAGE_ID, boot_volume_size_in_gbs=BOOT_VOLUME_GB ), create_vnic_details=oci.core.models.CreateVnicDetails( subnet_id=SUBNET_ID, assign_public_ip=False # Routed internally via our Gateway node ), metadata={ \u0026#34;ssh_authorized_keys\u0026#34;: SSH_PUBLIC_KEY } ) # 3. Continuous Provisioning Loop attempt = 1 delay_seconds = 30 # Polling interval print(f\u0026#34;🚀 Starting automated provisioning for {INSTANCE_NAME} ({OCPUS} OCPU / {MEMORY_IN_GBS}GB RAM)...\u0026#34;) while True: try: print(f\u0026#34;[{time.strftime(\u0026#39;%Y-%m-%d %H:%M:%S\u0026#39;)}] Attempt #{attempt}: Requesting VM.Standard.A1.Flex...\u0026#34;) response = compute_client.launch_instance(launch_details) instance_id = response.data.id print(f\u0026#34;\\n🎉 SUCCESS! Instance provisioned successfully!\u0026#34;) print(f\u0026#34;📋 Instance OCID: {instance_id}\u0026#34;) break except oci.exceptions.ServiceError as e: if e.status == 500 and \u0026#34;Out of host capacity\u0026#34; in e.message: print(f\u0026#34; ⏳ Capacity full. Retrying in {delay_seconds}s...\u0026#34;) elif e.status == 429: print(f\u0026#34; ⚠️ Rate limited (HTTP 429). Backing off for 60s...\u0026#34;) time.sleep(30) else: print(f\u0026#34; ❌ Unexpected Error [{e.status}]: {e.message}\u0026#34;) time.sleep(delay_seconds) attempt += 1 if __name__ == \u0026#34;__main__\u0026#34;: launch_arm_instance() 💡 4 Essential Tips for Maximum Success 1. Don\u0026rsquo;t Request All 4 OCPUs / 24 GB in a Single Chunk While Oracle allows up to 4 OCPUs and 24 GB RAM for free, requesting a smaller shape (such as 2 OCPUs and 12 GB RAM) has a substantially higher probability of succeeding because smaller memory blocks are freed up more frequently. You can always resize the instance later or run a secondary worker! 2. Run the Script on an Existing Free Micro Instance Instead of running the script on your local laptop (which goes to sleep when closed), run the provisioning script in a tmux or systemd background session on an existing AMD64 E2.1.Micro (amd10) instance within the same Oracle Cloud region. This ensures 24/7 uninterrupted execution with 0ms regional network latency to OCI API endpoints! 3. Sizing Your Boot Volume Oracle Cloud gives you 200 GB of free total boot volume storage. Allocate 50 GB to each of your two AMD instances and 50–100 GB to your ARM instance to stay comfortably within the 200 GB limit. 4. Keep VNIC Private if Using an Ingress Gateway If you run a dedicated edge gateway (like our amd10 running Caddy), set assign_public_ip=False on the ARM instance. This saves your 1 free public IPv4 address for your edge gateway and routes internal traffic securely across the Oracle VCN private subnet (10.0.0.0/16). 🏆 The Result: Our arm10 Powerhouse is Live! Using this exact method, our script caught a capacity window and successfully spun up arm10:\nArchitecture: aarch64 Ampere Altra Specs: 2 OCPUs / 12 GB RAM / 50 GB NVMe Storage Role: K3s Kubernetes Control Plane, PostgreSQL 18.6 with pgvector, AnythingLLM Document RAG, and Memos AI Journal. With a little patience and automation, you can unlock enterprise-grade cloud compute completely for free! 🚀☁️\n","permalink":"https://vinhthang.dev/posts/snagging-oracle-always-free-arm-ampere/","summary":"\u003cp\u003eIf you have ever tried to launch an \u003cstrong\u003eARM64 Ampere A1 Flex\u003c/strong\u003e compute instance on \u003cstrong\u003eOracle Cloud Infrastructure (OCI) Always Free\u003c/strong\u003e, you have almost certainly encountered this dreaded error message:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003ccode\u003e500-InternalError: Out of host capacity for shape VM.Standard.A1.Flex in availability domain...\u003c/code\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eOracle\u0026rsquo;s Always Free tier offers one of the most generous compute allowances in the industry—\u003cstrong\u003eup to 4 OCPUs and 24 GB RAM\u003c/strong\u003e on enterprise-grade Ampere Altra ARM64 processors. Because of this, capacity in popular data centers (like Tokyo, Seoul, Ashburn, Frankfurt, and Singapore) is constantly saturated.\u003c/p\u003e","title":"How to Snag the Elusive Oracle Cloud Always Free ARM64 Ampere A1 Instance"},{"content":"Most self-hosted AI setups stop at running a basic chat UI connected to a remote API. But what if your self-hosted AI wasn\u0026rsquo;t just a conversational bot, but a fully empowered autonomous system agent capable of managing your infrastructure, installing command-line tools, debugging services, and querying custom vector databases?\nIn this post, I will explore our complete self-hosted AI architecture running on an Always Free cloud Kubernetes cluster—featuring AnythingLLM, PostgreSQL 18 with pgvector, and the Google Antigravity CLI (agy).\n🏗️ The End-to-End AI Stack Our AI stack is organized into three complementary layers:\n┌─────────────────────────────────────────────────────────────────────────┐ │ 1. User Interaction Layer │ │ • AnythingLLM Web UI (https://ai.vinhthang.dev) │ │ • Document RAG, PDF Analysis, Workspace Memory, Voice Synthesizer │ └────────────────────────────────────┬────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────┐ │ 2. System Agent Core │ │ • Antigravity CLI (`agy`) Daemon \u0026amp; OpenAI-Compatible Gateway │ │ • Autonomous Terminal Execution (install git, compile, run scripts) │ │ • Tool Calling, Multi-Agent Delegation, and Real-Time Reasoning │ └────────────────────────────────────┬────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────┐ │ 3. High-Performance Data Layer │ │ • PostgreSQL 18.6 with `pgvector 0.8.6` │ │ • Asynchronous I/O (AIO) for Ultra-Fast Vector Cosine Similarity │ │ • LanceDB Embedded Fast Local Storage │ └─────────────────────────────────────────────────────────────────────────┘ 🧠 1. AnythingLLM: The Unified Knowledge \u0026amp; RAG Hub For interacting with complex technical specifications, architecture diagrams, and research papers, we run AnythingLLM in Kubernetes on our ARM64 powerhouse node (arm10 with 12 GB RAM):\nDocument Chunking \u0026amp; Vector Search: Ingests multi-page PDFs, markdown wikis, and codebases, converting them into semantic vector embeddings. Granular Workspaces: Isolate context across different domains (e.g. DevOps, Backend Services, Personal Notes). Privacy \u0026amp; Security: Enforced with a Centralized Google OAuth2 Single Sign-On (SSO) gateway, allowing only whitelisted administrative accounts to access the portal. 🛠️ 2. The Powerhouse: Antigravity CLI (agy) as an Autonomous System Agent The true differentiator in our setup is integrating the Google Antigravity CLI (agy).\nUnlike conventional language models that can only output text into a chatbox, agy is equipped with deep host-level execution capabilities:\n⚡ What the agy Agent Can Do on the Host: Package \u0026amp; Tool Installation: When tasked with setting up a project, agy can automatically update system repositories and install packages like git, docker, rustc, ripgrep, or kubectl without requiring manual terminal babysitting. Command Execution \u0026amp; Terminal Automation: It can compile codebases, run test suites, inspect system logs with journalctl, and diagnose container failures in real time. Infrastructure \u0026amp; Kubernetes Management: The agent can craft Kubernetes YAML manifests, apply them to the cluster via kubectl, verify pod health, and perform rolling updates. Autonomous Problem Solving: When an error occurs (such as an unreachable port or a missing dependency), agy inspects firewall rules, checks listening ports with ss -tulpn, edits configuration files, and restarts services autonomously. # Running Antigravity CLI in automated headless mode agy -p \u0026#34;Inspect the cluster pods and verify all services are healthy\u0026#34; --dangerously-skip-permissions To allow AnythingLLM to query agy as an AI model, we built a lightweight FastAPI OpenAI Proxy (agy-proxy) running as a native systemd service on the server. AnythingLLM simply sends standard /v1/chat/completions requests to http://localhost:8000/v1, and the proxy delegates execution directly to the authenticated Antigravity engine!\n🐘 3. PostgreSQL 18.6 with pgvector 0.8.6: Next-Gen Vector Storage Relational data and AI vector embeddings converge in our PostgreSQL 18.6 instance:\nNative Asynchronous Direct I/O (AIO): PostgreSQL 18 introduces true asynchronous disk reads, drastically reducing latency during high-dimensional vector nearest-neighbor searches (HNSW / IVFFlat). 64-bit Transaction IDs: Eliminates transaction wraparound freezes, ensuring uninterrupted operation for high-frequency database writes. Unified State Store: Powering our Memos AI Journal (memos.vinhthang.dev) and vector collections under a single, highly optimized relational engine. 🚀 The Advantage of Self-Hosted Hybrid AI Capability Standard Web Chatbot Our Autonomous AI Stack Document Understanding Limited to small uploads Hundreds of pages via PostgreSQL vector RAG System Administration ❌ None (Text only) ✅ Installs packages, runs shell commands, edits files Cloud Orchestration ❌ None ✅ Manages Kubernetes, GitOps, and Docker containers Data Privacy Third-party cloud storage 🔒 Self-hosted on private NVMe disk with Google SSO Cost Expensive monthly tiers ♾️ 100% Free on Oracle Cloud Infrastructure 🎯 Conclusion By bridging AnythingLLM with the autonomous capabilities of the Antigravity CLI (agy) and the speed of PostgreSQL 18 + pgvector, we\u0026rsquo;ve transformed a simple cloud server into an intelligent, self-healing developer companion that can build, maintain, and expand its own environment.\nStay tuned for more deep dives into autonomous AI and cloud architecture on vinhthang.dev!\n","permalink":"https://vinhthang.dev/posts/self-hosted-ai-agent-architecture/","summary":"\u003cp\u003eMost self-hosted AI setups stop at running a basic chat UI connected to a remote API. But what if your self-hosted AI wasn\u0026rsquo;t just a conversational bot, but a \u003cstrong\u003efully empowered autonomous system agent\u003c/strong\u003e capable of managing your infrastructure, installing command-line tools, debugging services, and querying custom vector databases?\u003c/p\u003e\n\u003cp\u003eIn this post, I will explore our complete self-hosted AI architecture running on an Always Free cloud Kubernetes cluster—featuring \u003cstrong\u003eAnythingLLM\u003c/strong\u003e, \u003cstrong\u003ePostgreSQL 18 with \u003ccode\u003epgvector\u003c/code\u003e\u003c/strong\u003e, and the \u003cstrong\u003eGoogle Antigravity CLI (\u003ccode\u003eagy\u003c/code\u003e)\u003c/strong\u003e.\u003c/p\u003e","title":"Autonomous AI Infrastructure: AnythingLLM, PostgreSQL 18 pgvector, and the Antigravity CLI (agy) System Agent"},{"content":"How do you turn 3 separate Always Free cloud virtual machines into a resilient, production-grade, multi-architecture Kubernetes cluster capable of running high-performance relational databases, AI vector stores, lossless music streaming, and personal cloud storage?\nIn this post, I will break down the complete architecture of our newly deployed 3-Node Hybrid K3s Cluster running on Oracle Cloud Infrastructure (OCI) under our custom domain vinhthang.dev.\n🏛️ The 3-Node Cluster Topology Oracle Cloud\u0026rsquo;s Always Free tier offers an incredible set of compute resources:\n2x AMD64 E2.1.Micro instances (1 OCPU, 1 GB RAM each) 1x ARM64 Ampere A1 instance (flexibly sized up to 4 OCPUs and 24 GB RAM; here sized at 2 OCPUs, 12 GB RAM) 1x Reserved Static Public IPv4 Address Instead of treating these machines as isolated Docker boxes, we unified them into a cohesive fleet:\n[ 🌐 Internet Traffic (*.vinhthang.dev) ] │ ▼ ┌─────────────────────────────────────┐ │ amd10 (Edge Ingress Gateway) │ │ Edge Domain: vinhthang.dev │ │ • Caddy Reverse Proxy \u0026amp; Let\u0026#39;s Encrypt│ │ • AdGuard Home DNS-over-TLS (853) │ │ • Hugo PaperMod Tech Blog Engine │ └──────────────────┬──────────────────┘ │ Private Oracle VCN (10.0.0.0/16) ┌────────────────────────┴────────────────────────┐ │ │ ▼ ▼ ┌───────────────────────────────────────┐ ┌───────────────────────────────────────┐ │ arm10 (K3s Control Plane) │ │ amd11 (K3s Worker Node) │ │ ARM64 Ampere A1 (2 OCPU / 12GB) │ │ AMD64 E2.1.Micro (1 OCPU / 1GB) │ │ • K3s Control Plane \u0026amp; Scheduler │ │ • K3s Agent / Worker │ │ • PostgreSQL 18.6 + pgvector 0.8.6 │ │ • Navidrome Music (45 GB FLAC disk) │ │ • AnythingLLM AI Agent \u0026amp; Vector RAG │ │ • FileBrowser Cloud Storage │ │ • Memos AI Digital Notebook │ │ • VietCalendar High-Speed Rust API │ │ • Central Google OAuth2 SSO Gateway │ └───────────────────────────────────────┘ └───────────────────────────────────────┘ 🚀 Key Architectural Innovations 1. Heterogeneous Multi-Architecture Scheduling (arm64 + amd64) Our K3s cluster seamlessly schedules containers across both CPU architectures using Kubernetes node selectors:\nARM64 Workloads (arm10): Heavy compute, AI vector processing, and memory-intensive databases (PostgreSQL 18, AnythingLLM, Memos, oauth2-proxy). AMD64 Workloads (amd11): I/O-bound storage services and x86-native binaries (Navidrome, FileBrowser, VietCalendar). # Example: Targeting AMD64 storage node in Kubernetes spec: nodeSelector: kubernetes.io/hostname: amd11 kubernetes.io/arch: amd64 2. State-of-the-Art PostgreSQL 18.6 with pgvector 0.8.6 At the heart of the cluster lies PostgreSQL 18.6 paired with pgvector 0.8.6 running natively on arm10:\nAsynchronous Direct I/O (AIO): Non-blocking disk reads for fast vector retrieval. 64-bit Transaction IDs: Eliminates transaction wraparound freezes permanently. Unified Database Stores: Backs Memos (memos), user analytics (vietcalendar), and custom AI embedding collections (vector_db). 3. Centralized Google OAuth2 Single Sign-On (SSO) Rather than placing separate auth proxies on every microservice, we built a Central Forward-Auth Gateway (auth.vinhthang.dev):\nCaddy checks session cookies with forward_auth against oauth2-proxy running in K3s. A single Google login issues a shared wildcard cookie for .vinhthang.dev. Logging in once unlocks files.vinhthang.dev, ai.vinhthang.dev, and memos.vinhthang.dev with zero repeated prompts. 4. Lossless Audio \u0026amp; Cloud Storage Preserved via hostPath Our 45 GB lossless FLAC music library and persistent storage were preserved without data migration by leveraging Kubernetes hostPath volume mounts on amd11:\nNavidrome directly streams from /opt/navidrome/music. FileBrowser provides web-based file management over /srv/Music. 🛠️ Unified Multi-Cluster Management From our development workstation, we unified management across 50 Kubernetes clusters inside a single ~/.kube/config:\noci-k3s (Our Oracle Cloud fleet) asgard.local \u0026amp; unicron.local (Local bare-metal clusters) 47 live Google Kubernetes Engine (GKE) clusters across enterprise projects. # Switch to Oracle Cloud K3s cluster in 1 command: kubectl get nodes -o wide --context oci-k3s 🌐 Live Production Service Directory All services are publicly live under our newly registered custom domain:\nService Endpoint Description 📝 Tech Blog vinhthang.dev Static Hugo site served with Brotli/Zstandard 🤖 AnythingLLM ai.vinhthang.dev Autonomous AI Agent \u0026amp; Document RAG 📝 Memos memos.vinhthang.dev PostgreSQL-backed AI Journal \u0026amp; Notes 🎵 Navidrome music.vinhthang.dev Subsonic FLAC Lossless Music Streaming 📁 FileBrowser files.vinhthang.dev Google SSO Cloud Storage 🔌 VietCalendar API api.vinhthang.dev Axum Rust Lunar Calendar Engine 🛡️ AdGuard Home adguard.vinhthang.dev Private DNS-over-TLS (Port 853) Co-authored with prompt engineering and pair-programming using Google Antigravity.\n","permalink":"https://vinhthang.dev/posts/hybrid-k3s-kubernetes-architecture-2026/","summary":"\u003cp\u003eHow do you turn 3 separate Always Free cloud virtual machines into a resilient, production-grade, multi-architecture Kubernetes cluster capable of running high-performance relational databases, AI vector stores, lossless music streaming, and personal cloud storage?\u003c/p\u003e\n\u003cp\u003eIn this post, I will break down the complete architecture of our newly deployed \u003cstrong\u003e3-Node Hybrid K3s Cluster\u003c/strong\u003e running on \u003cstrong\u003eOracle Cloud Infrastructure (OCI)\u003c/strong\u003e under our custom domain \u003ca href=\"https://vinhthang.dev\"\u003e\u003cstrong\u003e\u003ccode\u003evinhthang.dev\u003c/code\u003e\u003c/strong\u003e\u003c/a\u003e.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"-the-3-node-cluster-topology\"\u003e🏛️ The 3-Node Cluster Topology\u003c/h2\u003e\n\u003cp\u003eOracle Cloud\u0026rsquo;s Always Free tier offers an incredible set of compute resources:\u003c/p\u003e","title":"Architecting a Multi-Node Hybrid K3s Kubernetes Fleet on Oracle Cloud Always Free"},{"content":"Welcome to my personal tech blog!\nThis blog is built using Hugo with the PaperMod theme, served directly by Caddy with automatic Let\u0026rsquo;s Encrypt SSL.\n🌐 What is running on this server: AdGuard Home: Dual-stack network-wide ad \u0026amp; malware blocking (IPv4, IPv6, DoT). VietCalendar: High-performance Rust Lunar Calendar REST API \u0026amp; MCP Server. Navidrome: Bit-perfect lossless FLAC music streaming. FileBrowser: Private cloud storage with Google SSO. Hugo Blog: Ultra-fast static site serving at sub-millisecond speeds with 0 MB RAM overhead! Built with ❤️ on Oracle Linux 10 (amd10 in Tokyo).\n","permalink":"https://vinhthang.dev/posts/welcome/","summary":"\u003cp\u003eWelcome to my personal tech blog!\u003c/p\u003e\n\u003cp\u003eThis blog is built using \u003cstrong\u003e\u003ca href=\"https://gohugo.io/\"\u003eHugo\u003c/a\u003e\u003c/strong\u003e with the \u003cstrong\u003e\u003ca href=\"https://github.com/adityatelange/hugo-PaperMod\"\u003ePaperMod\u003c/a\u003e\u003c/strong\u003e theme, served directly by \u003cstrong\u003eCaddy\u003c/strong\u003e with automatic Let\u0026rsquo;s Encrypt SSL.\u003c/p\u003e\n\u003ch3 id=\"-what-is-running-on-this-server\"\u003e🌐 What is running on this server:\u003c/h3\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eAdGuard Home\u003c/strong\u003e: Dual-stack network-wide ad \u0026amp; malware blocking (IPv4, IPv6, DoT).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eVietCalendar\u003c/strong\u003e: High-performance Rust Lunar Calendar REST API \u0026amp; MCP Server.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNavidrome\u003c/strong\u003e: Bit-perfect lossless FLAC music streaming.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFileBrowser\u003c/strong\u003e: Private cloud storage with Google SSO.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHugo Blog\u003c/strong\u003e: Ultra-fast static site serving at sub-millisecond speeds with \u003cstrong\u003e0 MB RAM overhead\u003c/strong\u003e!\u003c/li\u003e\n\u003c/ol\u003e\n\u003chr\u003e\n\u003cp\u003e\u003cem\u003eBuilt with ❤️ on Oracle Linux 10 (\u003ccode\u003eamd10\u003c/code\u003e in Tokyo).\u003c/em\u003e\u003c/p\u003e","title":"🚀 Welcome to my Self-Hosted Cloud on Oracle Always Free"}]