Cluster Health Monitoring design¶
A design for surfacing kensan-lab's health (node liveness, resources, endpoint reachability) in a single Grafana view and notifying Slack on anomalies. Builds in three phases on top of the existing observability stack (observability.md).
Background and gaps¶
The existing stack already has the data path and alert path in place:
- node-exporter (DaemonSet) + kube-state-metrics → Prometheus (7d retention)
- kube-prometheus-stack's defaultRules (
node/nodeExporterAlertingenabled) → NodeNotReady and similar already notify Slack's#k8s-alerts - Grafana's sidecar auto-imports ConfigMap dashboards (existing: controlplane / longhorn / claude-code / otel-apm)
What's missing:
| # | Gap | Phase |
|---|---|---|
| 1 | No single dashboard for overall cluster health (node liveness, temperature, remaining microSD capacity, wired/WiFi state) | Phase 1 |
| 2 | Liveness monitoring only means "could Prometheus scrape it." Can't distinguish a dead kubelet from a dead OS, or a wired-link drop from a total outage. HTTP reachability of each UI through the Gateway is also unmonitored | Phase 2 |
| 3 | The monitoring stack itself is an m4neo SPOF. Prometheus / Alertmanager / Grafana all require hardware-class=high-performance, which pins them to m4neo. If m4neo dies, monitoring goes silent along with it — no alert fires |
Phase 3 |
Phase 1 — Cluster Health dashboard¶
No deployment changes. Adds a single dashboard using only existing metrics.
- Location:
kubernetes/observability/grafana/resources/cluster-health-dashboard.yaml(a ConfigMap withgrafana_dashboard: "1"+grafana_folder: "Platform"; same shape as the controlplane-dashboard) - Approach: hand-built rather than importing a community dashboard (e.g. Node Exporter Full) — tuned to a density where a 4-node homelab is legible at a glance
Panel layout¶
Row 1: Node liveness
| Panel | Query | Type |
|---|---|---|
| Node Ready | kube_node_status_condition{condition="Ready",status="true"} |
stat (per node, green/red) |
| node-exporter up history | up{job="node-exporter"} |
state timeline |
| Node Uptime | node_time_seconds - node_boot_time_seconds |
stat |
| Pressure conditions | kube_node_status_condition{condition=~"MemoryPressure\|DiskPressure\|PIDPressure",status="true"} |
table / stat |
Row 2: Resources
| Panel | Query | Notes |
|---|---|---|
| CPU usage | 1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) |
Per-node timeseries |
| Memory usage | 1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes |
Per-node timeseries |
| Disk usage (root) | 1 - node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / node_filesystem_size_bytes{...} |
Primary purpose: catching microSD exhaustion on the Pis. Thresholds: yellow at 80%, red at 90% |
| CPU temperature | node_hwmon_temp_celsius (falls back to node_thermal_zone_temp if unavailable on the Pis) |
Watches for thermal throttling on the Pis (80°C threshold) |
Row 3: Network (visualizing the wired/WiFi dual-path setup)
| Panel | Query | Notes |
|---|---|---|
| Wired link state | node_network_carrier{device=~"eth0\|eno1\|enp4s0"} |
Detects a wired drop that fell back to WiFi |
| WiFi link state | node_network_carrier{device=~"wlan0\|wlp3s0"} |
|
| Interface traffic | rate(node_network_receive_bytes_total{device=~"eth.*\|en.*\|wlan.*\|wlp.*"}[5m]) |
Shows which path traffic is actually flowing over |
Row 4: Cluster overview
| Panel | Query | Notes |
|---|---|---|
| Abnormal pod count | sum(kube_pod_status_phase{phase=~"Pending\|Failed\|Unknown"}) |
stat, green at 0 |
| Container restarts | sum by(namespace)(increase(kube_pod_container_status_restarts_total[1h])) |
Early detection of CrashLoops |
| PVC usage | kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes |
Covers both Longhorn and local-path |
| Pods per node | sum by(node)(kube_pod_info) |
Checking for scheduling skew |
Things to confirm during implementation:
- Confirm the actual temperature metric name on the Pis (
node_hwmon_temp_celsiusvsnode_thermal_zone_temp) against live Prometheus - Confirm
node_network_carrieris available on the WiFi interfaces (fall back tonode_network_upif not)
Phase 2 — blackbox-exporter (probe-based liveness monitoring)¶
Adds active ICMP / HTTP probes on top of Prometheus's internal-view scraping.
- Location:
kubernetes/observability/blackbox-exporter/(config.json+values.yaml). Auto-discovered by the observability ApplicationSet (git file generator) - Chart:
prometheus-community/prometheus-blackbox-exporter - Note: ICMP probes need the
NET_RAWcapability (granted via the chart'ssecurityContext)
Probe targets¶
Defined via the Prometheus Operator's Probe CRD (placed in resources/):
| Type | Target | What it reveals |
|---|---|---|
| ICMP | Wired IPs: 192.168.0.107–110 | OS-level liveness (distinguishes this from a dead kubelet) |
| ICMP | WiFi IPs: 192.168.0.207–210 | Whether the WiFi fallback is alive when the wired link drops |
| HTTP | https://{grafana,argocd,backstage,vault,prometheus}.platform.yu-min3.com |
End-to-end reachability through Gateway → cert → backend |
New alerts (PrometheusRule)¶
| Alert | Condition | Severity |
|---|---|---|
| NodeICMPUnreachable | Both wired and WiFi ICMP probes fail, sustained 5 minutes | critical |
| NodeWiredLinkDown | Wired probe fails & WiFi probe succeeds, sustained 10 minutes | warning (running on fallback) |
| PlatformEndpointDown | HTTP probe fails, sustained 5 minutes | warning |
Add a probe-results row to the dashboard (update the Phase 1 JSON).
Phase 3 — Grafana Cloud remote_write (mitigating the m4neo SPOF)¶
Puts "monitoring the monitoring, plus a dashboard that stays visible during an outage" outside the cluster. Changed from the original plan of healthchecks.io (Watchdog forwarding only) to a filtered remote_write into Grafana Cloud's free tier — this adds a dead-man's-switch (no-data alert) plus the ability to "view metrics from just before a total cluster outage, from outside the cluster."
- Mechanism: local Prometheus scraping stays entirely in-cluster as before;
remote_write(outbound HTTPS push, no inbound needed) replicates a carefully-chosen subset of series to Mimir on Grafana Cloud (Tokyo region) - Free tier: 10k active series / 14-day retention / 3 users. The allowlist measures out to ~1.4k series (14%) in practice
- Implementation:
- Create a Grafana Cloud account + stack (manual, no credit card required). Issue a remote_write token (
metrics:writescope) into Vault static (secret/monitoring/grafana-cloud/remote-write) - Sync it into a
grafana-cloud-remote-writeSecret (monitoringnamespace) via ESO - Add the endpoint + basicAuth +
writeRelabelConfigstoprometheusSpec.remoteWrite(keep the cluster-health dashboard's series, drop non-idle CPU modes) - Replicate the Cluster Health dashboard on the Cloud side (only the datasource uid needs to change)
- Set up a no-data alert on the Cloud side (
up{job="node-exporter"}no longer arriving) — this is the dead-man's-switch - Failure modes covered: m4neo going down, Prometheus / Alertmanager stopping, a total cluster outage, home network / ISP outage
- Candidate Phase 4 (optional): an external HTTP probe via Cloudflare Workers cron + a public status page (to distinguish a home-ISP outage from a Gateway failure from an outside vantage point)
PR split¶
| PR | Content | Deploy impact |
|---|---|---|
| 1 | This design doc + Phase 1 dashboard | None (ConfigMap addition only) |
| 2 | Phase 2 blackbox-exporter + Probes + alerts + dashboard update | One new Pod (Deployment) |
| 3 | Phase 3 Grafana Cloud remote_write (ExternalSecret + values.yaml) | Prometheus config reload |
Related¶
- Overview of the existing stack: observability.md
- Node topology and labels:
.claude/rules/kubernetes-cluster.md - The wired/WiFi dual-path setup: network.md