Building an Ultra-Lightweight Cloud-Native Observability & Analytics Stack: Metrics, Logs, Traces & Status on OCI

Running a complex fleet of microservices—spanning AI document assistants, vector databases, lossless audio streamers, private DNS resolvers, and astronomical APIs—demands complete, real-time visibility. But traditional enterprise observability suites (Elasticsearch, heavy Prometheus clusters, DataDog) frequently devour gigabytes of memory and CPU cycles just to monitor a small infrastructure.

In this deep dive, I break down how we architected and deployed a complete, ultra-lightweight, 360-degree Observability, Logging, Analytics, and Uptime Stack across our 3-node hybrid cloud fleet at vinhthang.dev, maintaining sub-millisecond query latencies while consuming less than 250 MB total RAM!

🏛️ The 5 Pillars of Our Modern Telemetry Architecture

Instead of deploying bloated monolithic logging tools, we engineered a modular, decoupled architecture where each layer excels at its specific domain:

                                  [ 🌐 Public Traffic & Visitors ]
                                                 │
                                                 ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                   1. Edge Gateway (amd10)                                       │
│  • Caddy TLS Termination & Reverse Proxy                                                        │
│  • Central Google OAuth2 Single Sign-On (auth.vinhthang.dev)                                    │
│  • Privacy-Friendly Blog Analytics Snippet (analytics.vinhthang.dev)                           │
└──────────────────────────────────────────────┬──────────────────────────────────────────────────┘
                                               │ (Private Oracle VCN 10.0.0.0/16)
                                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                 2. Kubernetes Cluster (arm10 & amd11)                           │
│                                                                                                 │
│  📊 TSDB METRICS LAYER           🪵 LOGGING LAYER                🔭 DISTRIBUTED TRACING LAYER    │
│  • VictoriaMetrics (Port 8428)    • VictoriaLogs (Port 9428)      • OpenTelemetry Collector     │
│  • PromQL Compatible (< 35MB)     • Promtail Multi-Node Daemon    • OTLP gRPC (:4317) & HTTP    │
│  • 30-Day Auto Retention          • 15x Log Compression (< 40MB)  • Application Tracing Spans   │
│                                                                                                 │
│  🟢 WATCHDOG HEALTH LAYER        📈 VISITOR ANALYTICS LAYER      🐘 DATA PERSISTENCE LAYER      │
│  • Uptime Kuma 2.5.3 (Port 30002) • Umami Analytics (Port 30003)  • PostgreSQL 18.6 + pgvector  │
│  • 9 Cloud Monitors + Status Page • Relational Web Visitor DB     • AI Vectors & Memos Storage  │
└──────────────────────────────────────────────┬──────────────────────────────────────────────────┘
                                               │
                                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                               3. Unified Visualization Dashboard                                │
│                     📈 Grafana 11 Command Center (grafana.vinhthang.dev)                        │
│          • Google SSO Auto-Login • Multi-Engine DataSources (PromQL, LogsQL, SQL)               │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

📊 1. Metrics Layer: VictoriaMetrics (High-Performance PromQL TSDB)

Traditional Prometheus TSDBs can easily consume 500 MB to 1 GB of memory for indexing timeseries metrics. To keep our footprint microscopic, we selected VictoriaMetrics:

• RAM Footprint: < 35 MB RAM (10x lighter than Prometheus).
• Native PromQL: 100% drop-in compatible with standard Prometheus formulas and Grafana community dashboards.
• Storage Compression: Gorilla delta-of-delta encoding, achieving less than 1.5 bytes per metric datapoint with automated 30-day data retention.
• Automatic Scraping: VictoriaMetrics scrapes Uptime Kuma (:3001/metrics) and OpenTelemetry Collector every 30 seconds.

🪵 2. Logging Layer: VictoriaLogs & Multi-Node Promtail

For container and system log aggregation, we bypassed the heavy Elasticsearch/Loki memory footprint and deployed VictoriaLogs:

• Ultra-Lean Engine: VictoriaLogs uses only ~40 MB RAM on arm10.
• 15x Log Compression: Compresses raw JSON/CRI container logs by over 90%, storing millions of log lines in a few megabytes of disk.
• Multi-Node Promtail DaemonSet: Deployed across all nodes (arm10 and amd11) to automatically harvest and stream container logs from /var/log/pods/*/*/*.log.
• Intuitive LogsQL: Querying logs is blazing fast without index bloat:

  # Search all logs across pods
  *
  # Search specific microservice errors
  app:anythingllm AND error
  # Aggregate error rates
  error | stats count() by (container)
  

🟢 3. Watchdog & Status Layer: Uptime Kuma v2.5.3

For synthetic health monitoring and external availability testing, we upgraded to Uptime Kuma v2.5.3:

• Decoupled Architecture: Uses embedded SQLite with WAL mode to ensure the health monitor remains 100% operational even if the central database restarts.
• Continuous Probing: Probes all 9 microservices across HTTPS, TLS handshakes, HTTP response codes, and SSL certificate expiration countdowns.
• Public Status Page: Hosted at https://status.vinhthang.dev, secured via Central Google Single Sign-On for administrative control.

📈 4. Web & Visitor Analytics: Umami + PostgreSQL 18.6

Instead of injecting privacy-invasive third-party scripts (like Google Analytics), we self-host Umami Analytics:

• Cookieless & GDPR Compliant: 100% anonymous visitor tracking with zero user fingerprinting or personal data collection.
• PostgreSQL 18 Integration: All pageviews, referrers, devices, and geographic insights are stored directly in PostgreSQL 18.6 on arm10.
• Accessible via: https://analytics.vinhthang.dev.

📈 5. Command Center: Grafana 11 Single-Pane-of-Glass

To bring everything together, we deployed Grafana 11 at https://grafana.vinhthang.dev:

• Single Sign-On Auth Proxy: Automatically logs you in as Admin via Google SSO ([email protected]) with zero secondary password friction.
• Multi-Engine Telemetry Matrix:
  1. VictoriaMetrics (PromQL): Real-time service status, latency wave charts, SSL expiration gauges.
  2. VictoriaLogs (LogsQL): Live streaming container logs with full-text search.
  3. PostgreSQL 18 (SQL): Relational notes from Memos, tracked traffic events from Umami, and pgvector AI index health.

⚡ Real-World Benchmarking: 100 Concurrent Requests

To verify that our monitoring and stateless microservices can handle burst traffic, we subjected our VietCalendar Rust API (api.vinhthang.dev) to a 100-request concurrent load test through Cloudflare and Caddy:

• Total Requests: 100 / 100
• Success Rate: 100.00% 200 OK (0 errors)
• Average Latency: 801 ms (including end-to-end public TLS negotiation)
• Throughput: 10.3 requests/sec
• Memory Spike: 0 MB (Rust astronomical calculations executed purely in CPU registers).

📊 Complete Fleet Memory Footprint

Here is the actual memory allocation across the entire cloud fleet after deploying the full observability stack:

🚀 Key Takeaways

1. You Don’t Need Massive Cloud Budgets for Enterprise Observability: By combining VictoriaMetrics, VictoriaLogs, and Grafana, you achieve multi-node metrics, full-text logging, and distributed tracing in less than 250 MB RAM.
2. Decoupled Architecture Wins: Separating stateless APIs (Rust), synthetic watchdogs (Uptime Kuma), relational stores (PostgreSQL 18), and time-series TSDBs prevents cascading outages.
3. Single Sign-On at the Edge: Wrapping your monitoring dashboards in a central OAuth2 gateway guarantees security with zero login fatigue.

All Kubernetes manifests, Promtail configs, and Grafana dashboard definitions are open-source and codified in our Git repository! 🌟🥂