Secret Management¶
This platform manages secrets and encryption keys via four methods, chosen by use case.
Methods and when to use each¶
| Method | Use case | Rotation | Source of truth |
|---|---|---|---|
| Vault dynamic (VDBE) | Postgres app-connection creds that can tolerate a restart / credential reload | TTL 24h / 12h rotation | Dynamically generated by the Vault Database engine |
| Vault static (ESO) | DB root creds / API keys / admin passwords / assorted tokens | Write a new value to Vault; distributed within an hour at most | Vault KV v2 |
| Vault Transit | Encryption keys for PII stored in a DB (prior example: the old kensan's users.name) |
Manual rotation (vault write -f transit/keys/.../rotate); new encrypts always run against the latest key version |
Vault Transit engine (the key itself stays in Vault; only ciphertext lands in the DB) |
| SealedSecret | Vault's own startup dependencies, and creds for layers below Vault (cert-manager DNS-01 / Longhorn R2) | Manual (infrequent) | The Git repo |
Current state (2026-07): after removing kensan-legacy (#403/#404) and reverting Keycloak (ADR-019), there are currently zero active Vault dynamic / Transit instances. The rails (chart / mount / ApplicationSet) are already running — adding a consumer is just a single file under
platform-values/. Every secret currently in use is either Vault static (ESO) or a SealedSecret.
Policy¶
- Postgres app-connection creds where minimizing the blast radius of a leak matters most → Vault dynamic. It requires
envinjection + a Reloader restart, though, so it's never used for long-lived services that hold IdP / catalog-metastore / schema-owner responsibilities - Secrets that need rotation but only infrequently, and that can be fetched as long as Vault is up → Vault static (ESO)
- PII stored in a DB column (e.g. names) → Vault Transit (envelope encryption that never sends the key down to the app Pod or the DB — Stage 6)
- Vault itself / cert-manager / Longhorn backups, i.e. low-level dependencies that can't depend on Vault → SealedSecret (Cilium's internal PKI used to live here too, but has since migrated to cert-manager
Certificatemanagement — seekubernetes/network/cilium/resources/cilium-ca-certificate.yaml)
VCO coverage and exceptions¶
Declarative management on the Vault side is handled by the Vault Config Operator (VCO, redhat-cop/vault-config-operator). Its CRs live in the kensan-lab repo for GitOps; the controller reconciles them into Vault. Not every Vault API has a corresponding CR, though — a small number of exceptions run via a manual setup script.
Operations VCO can express declaratively¶
| Vault operation | CR | Where it's used |
|---|---|---|
| Secret engine mount | SecretEngineMount |
vault-database-engine/shared/mount.yaml, vault-transit-engine/shared/mount.yaml |
| Policy (HCL) | Policy |
vault-database-engine/shared/policy-eso-read.yaml, vault-transit-engine/chart/templates/policy.yaml |
| K8s auth role (binds an SA + grants a policy) | KubernetesAuthEngineRole |
Both engines' chart/templates/vault-auth-role.yaml |
| OIDC auth role | JWTOIDCAuthEngineRole |
Keycloak SSO integration (Stage 4) |
| Database engine config (root creds + connectionURL) | DatabaseSecretEngineConfig |
vault-database-engine/chart/templates/connection.yaml (root creds are referenced from Vault KV — fully declarative) |
| Database engine role (CREATE/REVOKE SQL) | DatabaseSecretEngineRole |
vault-database-engine/chart/templates/role.yaml |
| Identity Group / Alias | Group / GroupAlias |
The OIDC login UX (Stage 4-5) |
Stages 2 through 5, plus the policy/auth roles for Stage 6, are almost entirely handled by VCO.
Exceptions (run via a setup script)¶
| Vault operation | Situation | Handling |
|---|---|---|
| Creating / rotating a Transit key | VCO has no TransitSecretEngine-style CR |
Run kubernetes/secrets/vault-transit-engine/bootstrap/setup-transit-keys.sh once per new key name. Rotation is a manual vault write -f transit/keys/<name>/rotate |
OIDC config (auth/oidc/config) |
VCO's patch support doesn't cover this; the provider URL etc. requires a full write | Handled by bootstrap/vault/post-bootstrap-oidc-ux.sh (Stage 4) as a full write |
The reasoning for keeping this hybrid rather than removing it (including the trigger line for reconsidering that) is covered in ADR-015.
Inventory¶
Vault dynamic (Postgres app creds)¶
No active instances currently (vault-database-engine/platform-values/vault-database/
is empty). The last consumer, dagster, was removed along with kensan-legacy (#403);
Keycloak reverted to static (ADR-019).
The rails themselves (chart + shared + ApplicationSet) are already running — see the
Operations section below and
vault-database-engine/README.md
for how to add a consumer.
Vault static (via ESO, Vault KV v2 → K8s Secret, refreshInterval 1h)¶
| namespace | Secret | Contents | Role |
|---|---|---|---|
| argocd | argocd-oidc-secret | client-id / client-secret | Argo CD's Keycloak OIDC client creds (security/argocd/oidc) |
| auth-system | oauth2-proxy-secret | client-id / client-secret / cookie-secret | oauth2-proxy for Gateway ext_authz (platform-auth/istio-gateway/platform) |
| backstage | backstage-secret | GITHUB_TOKEN | Backstage's GitHub integration (a read-only PAT) |
| backstage | postgresql-secret | POSTGRES_USER/PASSWORD/DB | Postgres StatefulSet boot + the Backstage app (kept static — see the table below) |
| cloudflare-tunnel | cloudflare-tunnel | tunnel token | Cloudflare Tunnel |
| monitoring | alertmanager-slack | Slack webhook | Alertmanager notifications |
| monitoring | grafana-admin-secret | admin-user / admin-password | Grafana admin |
| monitoring | grafana-oidc-secret | GF_AUTH_GENERIC_OAUTH_CLIENT_ID/SECRET | Grafana's Keycloak OIDC client creds (observability/grafana/oidc) |
| monitoring | grafana-cloud-remote-write | username / password | Auth for Prometheus → Grafana Cloud remote-write (monitoring/grafana-cloud/remote-write) |
| platform-auth-prod | keycloak-secret | KEYCLOAK_ADMIN_PASSWORD / KC_DB_PASSWORD | Keycloak admin + DB password (KC_DB_PASSWORD is synced from platform-auth/prod/postgresql.POSTGRES_PASSWORD, ADR-019) |
| platform-auth-prod | postgresql-secret | POSTGRES_USER/PASSWORD/DB | Keycloak Postgres boot |
| {backstage, app-kensan} | ghcr-pull-secret | .dockerconfigjson (GHCR image pull token) |
imagePullSecrets in each namespace. Shares the Vault path secret/ghcr/pull-token (app-kensan gets it via the app-base chart's externalsecret-ghcr.yaml) |
| app-kensan | feed-google-drive | credentials.json / output-folder-id |
Google Drive read-only settings for the Daily Briefing Job (secret/app-kensan/feed-google-drive) |
(The old kensan namespace's 6 secrets and the shared app-prod ghcr-pull-secret
disappeared in 2026-07 along with the kensan-legacy removal / the per-app namespace migration.)
GHCR token READ/WRITE split (a standing convention)¶
The GHCR PAT is split into two, by scope:
| Kind | Scope | Where it lives | Where it's used |
|---|---|---|---|
| READ | read:packages (+ repo read) |
Vault / local .env READ_GHCR_PAT |
Image pulls (ghcr-pull-secret = Vault secret/ghcr/pull-token), Backstage's runtime GITHUB_TOKEN (Vault secret/backstage/github, for the url reader / catalog / TechDocs), local make dev/run |
| WRITE | write:packages |
Local .env WRITE_GHCR_PAT only |
Image pushes (make login / make push) |
- The WRITE token never lives in the cluster or Vault. Pushing is a build-machine-only operation; everything in-cluster only ever needs READ.
- Each Makefile (
backstage,apps/kensan, the scaffold template) usesWRITE_GHCR_PATfor login/push, andREAD_GHCR_PATfor the runtimeGITHUB_TOKEN. The oldGITHUB_GHCR_PATremains as a backward-compatible fallback.
Vault Transit (the app calls the encrypt/decrypt API directly)¶
Rather than being distributed as a K8s Secret, the app Pod logs into Vault via K8s SA auth and calls the Transit API directly to encrypt / decrypt. The key itself stays resident in Vault and never lands on the Pod or in the DB.
No active consumers currently
(vault-transit-engine/platform-values/vault-transit/ is empty). The prior example, the
users-name key (used to encrypt + HMAC the old kensan user-service's users.name
column, Stage 6), had its consumer instance removed along with kensan-legacy (#404).
The rails themselves are still live: both the Vault side (mount/policy/auth role) and the
consumer-namespace side (SA/ConfigMap) are already charted
(vault-transit-engine/chart/ + platform-values/vault-transit/<consumer>.yaml).
Consumer Deployments reference it via serviceAccountName + envFrom: configMapRef, with
rotation wired through Reloader. Only key creation is manual
(bootstrap/setup-transit-keys.sh — a VCO limitation; see "VCO coverage and exceptions"
above). Rotation is vault write -f transit/keys/<name>/rotate → transit/rewrap on the
app side.
Design details and how to add a consumer:
kubernetes/secrets/vault-transit-engine/README.md
SealedSecret (static, doesn't depend on Vault, low rotation frequency)¶
| namespace | Secret | Role | Why not Vault |
|---|---|---|---|
| vault | vault-aws-kms-credentials | KMS creds for Vault auto-unseal | Needed for Vault's own startup (avoids a circular dependency) |
| cert-manager | route53-credentials | IAM for Let's Encrypt DNS-01 | cert-manager is a low-level dependency, rotated infrequently |
| longhorn-system | longhorn-r2-backup | R2 backup token | Longhorn is self-contained |
(Cilium / Hubble's internal PKI used to be a SealedSecret too, but couldn't be taken over
due to an ownerRef conflict with the chart's own Secret, so it migrated to cert-manager
Certificate management — see the header comment in
kubernetes/network/cilium/resources/cilium-ca-certificate.yaml)
Kept static (deliberately not moved to dynamic)¶
| App | Reason | Policy |
|---|---|---|
| keycloak | A DB-credential-rotation-triggered Pod restart destroys SSO sessions. Keycloak is the core of the IdP / SSO and carries a large circular dependency (history of moving to dynamic and then reverting: ADR-019) | Pinned to the Vault static (ESO) keycloak-secret |
| backstage | A long-lived service where a 12h restart buys little; also has plugin-DB / schema-owner complications | Pinned to the Vault static (ESO) postgresql-secret |
(The old kensan-family static entries — polaris / kensan-app — disappeared entirely along with the kensan-legacy removal (ADR-017))
Operations¶
Creating a SealedSecret¶
# 1. Create the raw secret (dry-run)
kubectl create secret generic <name> --namespace <ns> \
--from-literal=<key>=<value> --dry-run=client -o yaml > temp/<name>-raw.yaml
# 2. Encrypt it with kubeseal
kubeseal --format yaml < temp/<name>-raw.yaml > kubernetes/<cat>/<comp>/resources/<name>-sealed.yaml
# 3. Discard the raw file
rm temp/<name>-raw.yaml
# 4. commit + push → the Sealed Secrets controller decrypts it in-cluster → the Secret is created
temp/*-raw.yaml is already .gitignored (never commit it).
Adding a Vault static (ESO) secret¶
# 1. Write it into Vault KV v2
vault kv put secret/<path> KEY1=value1 KEY2=value2
# 2. Create the ExternalSecret manifest (example)
cat <<EOF > kubernetes/<cat>/<comp>/resources/<name>-external-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: <name>
namespace: <ns>
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: <k8s-secret-name>
creationPolicy: Owner
data:
- secretKey: KEY1
remoteRef: { key: <path>, property: KEY1 }
EOF
# 3. commit + push → ESO syncs it within the refresh interval
Adding a Vault dynamic (VDBE) secret¶
- Create
kubernetes/secrets/vault-database-engine/platform-values/vault-database/<name>.yaml(overridingns,rootOwner,keyMapping, etc.) - The ApplicationSet picks it up and auto-generates a
DatabaseSecretEngineRole(Vault) + anExternalSecret(K8s) - Details:
kubernetes/secrets/vault-database-engine/README.md
Related documents¶
- Sealed Secrets controller:
kubernetes/secrets/sealed-secrets/ - External Secrets Operator:
kubernetes/secrets/external-secrets/ - Vault:
kubernetes/secrets/vault/ - Vault Config Operator:
kubernetes/secrets/vault-config-operator/ - Vault Database engine + ESO integration:
kubernetes/secrets/vault-database-engine/ - Vault Transit engine (Stage 6):
kubernetes/secrets/vault-transit-engine/