Grafana Independent Deployment¶
Overview¶
Grafana has been separated from kube-prometheus-stack and independently deployed as the unified visualization layer for the entire Observability stack (Prometheus/Tempo/Loki).
It is managed by the observability ApplicationSet, which discovers each component from a per-component config.json and renders a Helm multi-source Application.
- ApplicationSet: kubernetes/argocd/applications/observability/applicationset.yaml (generator globs kubernetes/observability/*/config.json)
- Component config: kubernetes/observability/grafana/config.json (chart repo/version, namespace, hasResources)
- Helm values: kubernetes/observability/grafana/values.yaml
- Custom resources: kubernetes/observability/grafana/resources/
There is no longer a hand-written app.yaml for grafana — the Application CR is generated from config.json.
Key Grafana Values Settings¶
File: kubernetes/observability/grafana/values.yaml
admin.existingSecret: Retrieve admin credentials from Sealed Secretsidecar.datasources.enabled: Auto-discover datasources from ConfigMapssidecar.dashboards.enabled: Auto-discover dashboards from ConfigMapssidecar.datasources.defaultDatasourceEnabled: false: Disable the built-in Prometheus datasource
Admin Credentials (External Secrets → Vault)¶
The Grafana admin credentials are no longer a hand-sealed Secret. They are managed by the External Secrets Operator (ESO), which reads them from Vault and materializes the grafana-admin-secret that values.yaml references via admin.existingSecret.
- ExternalSecret CR:
kubernetes/observability/grafana/resources/external-secret.yaml - Pulls
secret/monitoring/grafana/adminfrom Vault via thevault-backendClusterSecretStore and createsgrafana-admin-secretinmonitoring. - OIDC client secret:
kubernetes/observability/grafana/resources/external-secret-oidc.yaml
Prerequisites:
- The shared ClusterSecretStore
vault-backendis healthy (kubernetes/secrets/external-secrets/resources/cluster-secret-store.yaml— ESO reads Vault via the sharedexternal-secretsrole /external-secrets-readpolicy created by the bootstrap Terraform; no per-app Vault role is needed for Grafana) - The KV entry
secret/monitoring/grafana/adminis populated (bootstrap/vault/migrate-secret.sh)
OTel Dashboard Updates¶
Script to fetch official dashboards from Grafana.com and update ConfigMaps:
Output: kubernetes/observability/grafana/resources/dashboards.yaml
Verification¶
Datasources¶
Grafana UI -> Configuration -> Data Sources should display: - Prometheus (default, exemplars enabled) - Tempo (tracing) - Loki (logs)
Dashboards¶
Grafana UI -> Dashboards -> Browse should display: - OpenTelemetry folder: OTel APM, OTel for HTTP Services - General folder: Kubernetes monitoring dashboards
Sidecar Logs¶
kubectl logs -n monitoring deployment/grafana -c grafana-sc-datasources --tail=10
kubectl logs -n monitoring deployment/grafana -c grafana-sc-dashboard --tail=10
Troubleshooting¶
Grafana Pod in CrashLoopBackOff¶
Symptom: Only one datasource per organization can be marked as default
Fix: Set sidecar.datasources.defaultDatasourceEnabled: false in values.yaml
Datasources Not Showing¶
ConfigMap requires the grafana_datasource: "1" label.
Dashboards Not Showing¶
ConfigMap requires the grafana_dashboard: "1" label.
File Structure¶
kubernetes/observability/grafana/
├── config.json # ApplicationSet input (chart/version/namespace/hasResources)
├── values.yaml # Helm values
└── resources/
├── datasources.yaml # Prometheus/Tempo/Loki datasource definitions
├── dashboards.yaml # OTel dashboard ConfigMap
├── claude-code-dashboard.yaml # additional dashboard ConfigMaps
├── cluster-health-dashboard.yaml
├── controlplane-dashboard.yaml
├── longhorn-dashboard.yaml
├── external-secret.yaml # Grafana admin credentials (External Secrets → Vault)
├── external-secret-oidc.yaml # OIDC client secret
└── httproute.yaml
kubernetes/observability/prometheus/
├── values.yaml # grafana.enabled: false
└── resources/
├── httproute-prometheus.yaml
└── grafana-sealed-secret.yaml