Monitoring platform and alert governance
Built the company's monitoring from scratch: Grafana, OpenSearch and Fluent Bit first, then Prometheus, Loki and Alertmanager, with views scoped per service team and alerts by email.
My scope
Architecture, build, alert rules, notifications and docs. From 2025 the second and third dashboard/alert revisions were done with AI agents, after making sure the data could be backed up and restored.
Approach
- Host and cluster metrics and logs in one place; each service team's query scope is enforced server-side.
- Six operations dashboards and five per service team, covering services, pods, logs, routes and storage.
Verification
Alert email format accepted by IT; access boundaries verified on 2026-10-13.
Results
| Metric | Value | Source | Evidence |
|---|---|---|---|
| Alerting rules / recording rules / scrape targets | 58 / 22 / 18 | Prometheus API, 2026-10-21 | System record |
| Related hours | 496 h | Timesheets since Jun 2022 | System record |
| Logged hours split: me / AI execution | me ≈422 h, AI execution ≈74 h (hours before 2025-02-22 count as mine) | Own estimate; hours since 2025-02-22 split by the period’s ratio; company-logged hours only | Own estimate |
Limits and next steps
Verification in this case covers alert format and access boundaries; long-run stability observation and backup-restore drills are the next step.
Tools
- Grafana
- Prometheus
- Loki
- Alertmanager
- Fluent Bit
- OpenSearch