Observability
One view of a fragmented infrastructure
Built with
- Grafana
- Prometheus
- Loki
- Tempo
- ClickHouse
- The problem
- Infrastructure was spread across several providers and servers, each with its own logs and metrics. Understanding the health of the whole system meant checking many places, and nobody had the full picture.
- What I did
- Centralized metrics, logs and distributed traces in Grafana and Prometheus, with Loki, Tempo and ClickHouse behind them. Standardized structured log lines across services, connected the separate networks so everything reports to one place, and wired critical alerts to Telegram.
- The outcome
- A single source of truth to diagnose incidents. Bringing a new server or service into monitoring became a routine step, and the stack runs in production today.