Secrets: Vault at the core, four delivery methods, one escape hatch¶
Secret management for everything confidential — DB credentials, API keys, TLS certificates, encryption keys — built around one idea: give every secret the lightest delivery method that still closes its blast radius, instead of forcing everything through one pipe.
Design thesis: Vault is the core, but Vault centralization is not a goal. A secret moves into Vault only where Vault adds unique value — dynamic credentials, envelope encryption — and stays on simpler rails where it doesn't. Everything Vault itself depends on lives below Vault, sealed in Git. Vault's own configuration is declarative (operator CRs in Git) with a short, exhaustive list of script-managed exceptions.
What you'll find here: how four delivery methods divide the secret inventory, where the bootstrap boundary sits (what must not depend on Vault), and how rotation reaches running pods end-to-end — a decision framework you can reuse when choosing "does this secret really need Vault?"
Components¶
| dir | role | namespace |
|---|---|---|
vault/ |
Vault server (HA 3 replicas, KMS auto-unseal; KV / Database / Transit / OIDC mounts) | vault |
vault-config-operator/ |
Configures Vault declaratively from K8s CRs (mounts, policies, auth roles) | vault-config-operator |
vault-database-engine/ |
Dynamic DB credentials — self-made chart + shared resources, per-DB instances generated by ApplicationSet | vault |
vault-transit-engine/ |
Encryption-as-a-service — keys, policies, and auth roles for column-level PII encryption | vault |
external-secrets/ |
ESO: the Vault → K8s Secret bridge (refreshInterval 1h) | external-secrets |
sealed-secrets/ |
Encrypted-in-Git secrets for everything that must exist before Vault | sealed-secrets |
cert-manager/ |
TLS certificates (Let's Encrypt DNS-01 wildcards) — deliberately independent of Vault | cert-manager |
reloader/ |
Watches Secret/ConfigMap changes → rolls affected workloads (closes the rotation loop) | reloader |
Four methods, one flow¶
Everything in the top half depends on Vault; the SealedSecret lane at the bottom exists so that Vault (and the cluster) can boot at all.
%%{init: {"flowchart": {"htmlLabels": true}} }%%
flowchart LR
subgraph VAULT["Vault | HA ×3, KMS auto-unseal"]
DBE["Database engine<br/>dynamic creds (TTL 24h)"]
KV["KV v2<br/>static secrets"]
TR["Transit engine<br/>encrypt / decrypt API"]
end
VCO["Vault Config Operator<br/>CRs in Git → reconcile"] -.->|declarative config| VAULT
ESO["External Secrets Operator"]
DBE --> ESO
KV -->|refresh ≤1h| ESO
ESO -->|K8s Secret| POD["app Pods (env)"]
REL["Reloader"] -.->|Secret changed → rollout| POD
TR -->|"API call per encrypt/decrypt<br/>(key never leaves Vault)"| APP["app Pod (SA auth)"]
GIT["Git (SealedSecret YAML)"] --> SSC["sealed-secrets controller"]
SSC -->|K8s Secret| BOOT["bootstrap layer:<br/>Vault KMS cred / cert-manager DNS-01<br/>Longhorn R2"]
classDef default fill:#26221D,stroke:#4A4232,color:#FCFAF6
classDef vault fill:#075985,stroke:#38BDF8,color:#FCFAF6
classDef bridge fill:#2F6B45,stroke:#7AC99A,color:#FCFAF6
classDef boot fill:#8A4B12,stroke:#FDBA74,color:#FCFAF6
class DBE,KV,TR vault
class ESO,REL,SSC bridge
class GIT,BOOT boot
| Method | Use for | Rotation |
|---|---|---|
| Vault dynamic (Database engine) | Postgres app creds where a restart / credential reload is acceptable | TTL 24h, generated per lease |
| Vault static (KV via ESO) | API keys, admin passwords, tokens — low churn but rotatable | write new value to Vault → delivered within 1h |
| Vault Transit | PII columns in app DBs (envelope encryption; only ciphertext touches the DB) | manual key rotate + rewrap |
| SealedSecret | Anything below Vault's dependency line — Vault's own KMS unseal cred, cert-manager DNS-01 IAM, Longhorn R2 token | manual, low frequency |
Current inventory (2026-07): the dynamic and Transit rails are deployed and were battle-tested by the previous kensan application, but their last consumers left with the kensan-legacy retirement (ADR-017) and the Keycloak revert (ADR-019) — zero active instances today; every live secret rides Vault static (ESO) or SealedSecret. Adding a consumer back is one
platform-values/file (engine README).
Design rationale¶
Three principles thread the whole design:
- Right-size, don't centralize. Moving a secret to Vault dynamic buys blast-radius reduction but costs a restart on every rotation — so IdPs and long-lived schema-owning services (Keycloak, Backstage) deliberately stay on Vault static. The same restraint applies to certificates: Vault PKI is intentionally not enabled while cert-manager + Let's Encrypt already finish the job (ADR-007).
- Declarative by default, scripts as a bounded exception. Vault mounts, policies, and auth roles are Vault Config Operator CRs living in Git — GitOps all the way into Vault, no
terraform apply, no state file full of secrets. Only two operations stay script-managed (Transit key creation, OIDC config full-write), and that exception list is exhaustive by contract (ADR-015). - The bootstrap layer must not depend on Vault. Whatever Vault needs to start (KMS auto-unseal cred), and whatever must exist before Vault is healthy (cert-manager's DNS-01 credential, Longhorn's backup target), is sealed into Git and decrypted in-cluster — breaking the circular dependency by construction. (Cilium's internal PKI, once on this list, moved to a cert-manager
Certificatewith auto-renewal.)
Concrete choices:
- Rotation is closed end-to-end. ESO refreshes the K8s Secret; Reloader sees the change and rolls the workload. A rotation in Vault reaches running pods with no human in the loop.
- Keycloak DB credentials are the stress test. First kept static (restart breaks SSO sessions — ADR-008), then moved to Vault dynamic with explicit cascade-risk mitigations (ADR-013), then reverted to static after operating it confirmed ADR-008's prediction — rotation restarts kept breaking SSO sessions (ADR-019). A worked example of the "right-size" principle being re-evaluated twice as operational evidence accumulated.
- Transit keeps keys out of pods and DBs. Apps authenticate with their ServiceAccount and call encrypt/decrypt; the key material never leaves Vault, so a DB dump or pod compromise yields only ciphertext.
- Vault's image tag is explicitly pinned — an upgrade of the secret core should never ride in on a chart bump unnoticed (ADR-011).
Related¶
- Full decision matrix, secret inventory, and operations (seal / rotate / debug):
docs/secret-management/index.md - ADRs: 007 No Vault PKI / 008 Keycloak DB credentials / 011 Vault version pinning / 013 Keycloak → Vault dynamic / 019 Keycloak → revert to static / 015 VCO + setup-script hybrid
- AI operational rules (seal workflow, file safety):
.claude/rules/security-secrets.md