Skip to content

Secret Management

This platform manages secrets and encryption keys via four methods, chosen by use case.

Methods and when to use each

Method Use case Rotation Source of truth
Vault dynamic (VDBE) Postgres app-connection creds that can tolerate a restart / credential reload TTL 24h / 12h rotation Dynamically generated by the Vault Database engine
Vault static (ESO) DB root creds / API keys / admin passwords / assorted tokens Write a new value to Vault; distributed within an hour at most Vault KV v2
Vault Transit Encryption keys for PII stored in a DB (prior example: the old kensan's users.name) Manual rotation (vault write -f transit/keys/.../rotate); new encrypts always run against the latest key version Vault Transit engine (the key itself stays in Vault; only ciphertext lands in the DB)
SealedSecret Vault's own startup dependencies, and creds for layers below Vault (cert-manager DNS-01 / Longhorn R2) Manual (infrequent) The Git repo

Current state (2026-07): after removing kensan-legacy (#403/#404) and reverting Keycloak (ADR-019), there are currently zero active Vault dynamic / Transit instances. The rails (chart / mount / ApplicationSet) are already running — adding a consumer is just a single file under platform-values/. Every secret currently in use is either Vault static (ESO) or a SealedSecret.

Policy

  • Postgres app-connection creds where minimizing the blast radius of a leak matters most → Vault dynamic. It requires env injection + a Reloader restart, though, so it's never used for long-lived services that hold IdP / catalog-metastore / schema-owner responsibilities
  • Secrets that need rotation but only infrequently, and that can be fetched as long as Vault is up → Vault static (ESO)
  • PII stored in a DB column (e.g. names) → Vault Transit (envelope encryption that never sends the key down to the app Pod or the DB — Stage 6)
  • Vault itself / cert-manager / Longhorn backups, i.e. low-level dependencies that can't depend on Vault → SealedSecret (Cilium's internal PKI used to live here too, but has since migrated to cert-manager Certificate management — see kubernetes/network/cilium/resources/cilium-ca-certificate.yaml)

VCO coverage and exceptions

Declarative management on the Vault side is handled by the Vault Config Operator (VCO, redhat-cop/vault-config-operator). Its CRs live in the kensan-lab repo for GitOps; the controller reconciles them into Vault. Not every Vault API has a corresponding CR, though — a small number of exceptions run via a manual setup script.

Operations VCO can express declaratively

Vault operation CR Where it's used
Secret engine mount SecretEngineMount vault-database-engine/shared/mount.yaml, vault-transit-engine/shared/mount.yaml
Policy (HCL) Policy vault-database-engine/shared/policy-eso-read.yaml, vault-transit-engine/chart/templates/policy.yaml
K8s auth role (binds an SA + grants a policy) KubernetesAuthEngineRole Both engines' chart/templates/vault-auth-role.yaml
OIDC auth role JWTOIDCAuthEngineRole Keycloak SSO integration (Stage 4)
Database engine config (root creds + connectionURL) DatabaseSecretEngineConfig vault-database-engine/chart/templates/connection.yaml (root creds are referenced from Vault KV — fully declarative)
Database engine role (CREATE/REVOKE SQL) DatabaseSecretEngineRole vault-database-engine/chart/templates/role.yaml
Identity Group / Alias Group / GroupAlias The OIDC login UX (Stage 4-5)

Stages 2 through 5, plus the policy/auth roles for Stage 6, are almost entirely handled by VCO.

Exceptions (run via a setup script)

Vault operation Situation Handling
Creating / rotating a Transit key VCO has no TransitSecretEngine-style CR Run kubernetes/secrets/vault-transit-engine/bootstrap/setup-transit-keys.sh once per new key name. Rotation is a manual vault write -f transit/keys/<name>/rotate
OIDC config (auth/oidc/config) VCO's patch support doesn't cover this; the provider URL etc. requires a full write Handled by bootstrap/vault/post-bootstrap-oidc-ux.sh (Stage 4) as a full write

The reasoning for keeping this hybrid rather than removing it (including the trigger line for reconsidering that) is covered in ADR-015.

Inventory

Vault dynamic (Postgres app creds)

No active instances currently (vault-database-engine/platform-values/vault-database/ is empty). The last consumer, dagster, was removed along with kensan-legacy (#403); Keycloak reverted to static (ADR-019). The rails themselves (chart + shared + ApplicationSet) are already running — see the Operations section below and vault-database-engine/README.md for how to add a consumer.

Vault static (via ESO, Vault KV v2 → K8s Secret, refreshInterval 1h)

namespace Secret Contents Role
argocd argocd-oidc-secret client-id / client-secret Argo CD's Keycloak OIDC client creds (security/argocd/oidc)
auth-system oauth2-proxy-secret client-id / client-secret / cookie-secret oauth2-proxy for Gateway ext_authz (platform-auth/istio-gateway/platform)
backstage backstage-secret GITHUB_TOKEN Backstage's GitHub integration (a read-only PAT)
backstage postgresql-secret POSTGRES_USER/PASSWORD/DB Postgres StatefulSet boot + the Backstage app (kept static — see the table below)
cloudflare-tunnel cloudflare-tunnel tunnel token Cloudflare Tunnel
monitoring alertmanager-slack Slack webhook Alertmanager notifications
monitoring grafana-admin-secret admin-user / admin-password Grafana admin
monitoring grafana-oidc-secret GF_AUTH_GENERIC_OAUTH_CLIENT_ID/SECRET Grafana's Keycloak OIDC client creds (observability/grafana/oidc)
monitoring grafana-cloud-remote-write username / password Auth for Prometheus → Grafana Cloud remote-write (monitoring/grafana-cloud/remote-write)
platform-auth-prod keycloak-secret KEYCLOAK_ADMIN_PASSWORD / KC_DB_PASSWORD Keycloak admin + DB password (KC_DB_PASSWORD is synced from platform-auth/prod/postgresql.POSTGRES_PASSWORD, ADR-019)
platform-auth-prod postgresql-secret POSTGRES_USER/PASSWORD/DB Keycloak Postgres boot
{backstage, app-kensan} ghcr-pull-secret .dockerconfigjson (GHCR image pull token) imagePullSecrets in each namespace. Shares the Vault path secret/ghcr/pull-token (app-kensan gets it via the app-base chart's externalsecret-ghcr.yaml)
app-kensan feed-google-drive credentials.json / output-folder-id Google Drive read-only settings for the Daily Briefing Job (secret/app-kensan/feed-google-drive)

(The old kensan namespace's 6 secrets and the shared app-prod ghcr-pull-secret disappeared in 2026-07 along with the kensan-legacy removal / the per-app namespace migration.)

GHCR token READ/WRITE split (a standing convention)

The GHCR PAT is split into two, by scope:

Kind Scope Where it lives Where it's used
READ read:packages (+ repo read) Vault / local .env READ_GHCR_PAT Image pulls (ghcr-pull-secret = Vault secret/ghcr/pull-token), Backstage's runtime GITHUB_TOKEN (Vault secret/backstage/github, for the url reader / catalog / TechDocs), local make dev/run
WRITE write:packages Local .env WRITE_GHCR_PAT only Image pushes (make login / make push)
  • The WRITE token never lives in the cluster or Vault. Pushing is a build-machine-only operation; everything in-cluster only ever needs READ.
  • Each Makefile (backstage, apps/kensan, the scaffold template) uses WRITE_GHCR_PAT for login/push, and READ_GHCR_PAT for the runtime GITHUB_TOKEN. The old GITHUB_GHCR_PAT remains as a backward-compatible fallback.

Vault Transit (the app calls the encrypt/decrypt API directly)

Rather than being distributed as a K8s Secret, the app Pod logs into Vault via K8s SA auth and calls the Transit API directly to encrypt / decrypt. The key itself stays resident in Vault and never lands on the Pod or in the DB.

No active consumers currently (vault-transit-engine/platform-values/vault-transit/ is empty). The prior example, the users-name key (used to encrypt + HMAC the old kensan user-service's users.name column, Stage 6), had its consumer instance removed along with kensan-legacy (#404).

The rails themselves are still live: both the Vault side (mount/policy/auth role) and the consumer-namespace side (SA/ConfigMap) are already charted (vault-transit-engine/chart/ + platform-values/vault-transit/<consumer>.yaml). Consumer Deployments reference it via serviceAccountName + envFrom: configMapRef, with rotation wired through Reloader. Only key creation is manual (bootstrap/setup-transit-keys.sh — a VCO limitation; see "VCO coverage and exceptions" above). Rotation is vault write -f transit/keys/<name>/rotate → transit/rewrap on the app side.

Design details and how to add a consumer: kubernetes/secrets/vault-transit-engine/README.md

SealedSecret (static, doesn't depend on Vault, low rotation frequency)

namespace Secret Role Why not Vault
vault vault-aws-kms-credentials KMS creds for Vault auto-unseal Needed for Vault's own startup (avoids a circular dependency)
cert-manager route53-credentials IAM for Let's Encrypt DNS-01 cert-manager is a low-level dependency, rotated infrequently
longhorn-system longhorn-r2-backup R2 backup token Longhorn is self-contained

(Cilium / Hubble's internal PKI used to be a SealedSecret too, but couldn't be taken over due to an ownerRef conflict with the chart's own Secret, so it migrated to cert-manager Certificate management — see the header comment in kubernetes/network/cilium/resources/cilium-ca-certificate.yaml)

Kept static (deliberately not moved to dynamic)

App Reason Policy
keycloak A DB-credential-rotation-triggered Pod restart destroys SSO sessions. Keycloak is the core of the IdP / SSO and carries a large circular dependency (history of moving to dynamic and then reverting: ADR-019) Pinned to the Vault static (ESO) keycloak-secret
backstage A long-lived service where a 12h restart buys little; also has plugin-DB / schema-owner complications Pinned to the Vault static (ESO) postgresql-secret

(The old kensan-family static entries — polaris / kensan-app — disappeared entirely along with the kensan-legacy removal (ADR-017))

Operations

Creating a SealedSecret

# 1. Create the raw secret (dry-run)
kubectl create secret generic <name> --namespace <ns> \
  --from-literal=<key>=<value> --dry-run=client -o yaml > temp/<name>-raw.yaml

# 2. Encrypt it with kubeseal
kubeseal --format yaml < temp/<name>-raw.yaml > kubernetes/<cat>/<comp>/resources/<name>-sealed.yaml

# 3. Discard the raw file
rm temp/<name>-raw.yaml

# 4. commit + push → the Sealed Secrets controller decrypts it in-cluster → the Secret is created

temp/*-raw.yaml is already .gitignored (never commit it).

Adding a Vault static (ESO) secret

# 1. Write it into Vault KV v2
vault kv put secret/<path> KEY1=value1 KEY2=value2

# 2. Create the ExternalSecret manifest (example)
cat <<EOF > kubernetes/<cat>/<comp>/resources/<name>-external-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: <name>
  namespace: <ns>
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: <k8s-secret-name>
    creationPolicy: Owner
  data:
    - secretKey: KEY1
      remoteRef: { key: <path>, property: KEY1 }
EOF

# 3. commit + push → ESO syncs it within the refresh interval

Adding a Vault dynamic (VDBE) secret

  1. Create kubernetes/secrets/vault-database-engine/platform-values/vault-database/<name>.yaml (overriding ns, rootOwner, keyMapping, etc.)
  2. The ApplicationSet picks it up and auto-generates a DatabaseSecretEngineRole (Vault) + an ExternalSecret (K8s)
  3. Details: kubernetes/secrets/vault-database-engine/README.md