Skip to content

Secrets: Vault at the core, four delivery methods, one escape hatch

Secret management for everything confidential — DB credentials, API keys, TLS certificates, encryption keys — built around one idea: give every secret the lightest delivery method that still closes its blast radius, instead of forcing everything through one pipe.

Design thesis: Vault is the core, but Vault centralization is not a goal. A secret moves into Vault only where Vault adds unique value — dynamic credentials, envelope encryption — and stays on simpler rails where it doesn't. Everything Vault itself depends on lives below Vault, sealed in Git. Vault's own configuration is declarative (operator CRs in Git) with a short, exhaustive list of script-managed exceptions.

What you'll find here: how four delivery methods divide the secret inventory, where the bootstrap boundary sits (what must not depend on Vault), and how rotation reaches running pods end-to-end — a decision framework you can reuse when choosing "does this secret really need Vault?"

Components

dir role namespace
vault/ Vault server (HA 3 replicas, KMS auto-unseal; KV / Database / Transit / OIDC mounts) vault
vault-config-operator/ Configures Vault declaratively from K8s CRs (mounts, policies, auth roles) vault-config-operator
vault-database-engine/ Dynamic DB credentials — self-made chart + shared resources, per-DB instances generated by ApplicationSet vault
vault-transit-engine/ Encryption-as-a-service — keys, policies, and auth roles for column-level PII encryption vault
external-secrets/ ESO: the Vault → K8s Secret bridge (refreshInterval 1h) external-secrets
sealed-secrets/ Encrypted-in-Git secrets for everything that must exist before Vault sealed-secrets
cert-manager/ TLS certificates (Let's Encrypt DNS-01 wildcards) — deliberately independent of Vault cert-manager
reloader/ Watches Secret/ConfigMap changes → rolls affected workloads (closes the rotation loop) reloader

Four methods, one flow

Everything in the top half depends on Vault; the SealedSecret lane at the bottom exists so that Vault (and the cluster) can boot at all.

%%{init: {"flowchart": {"htmlLabels": true}} }%%
flowchart LR
    subgraph VAULT["Vault | HA ×3, KMS auto-unseal"]
        DBE["Database engine<br/>dynamic creds (TTL 24h)"]
        KV["KV v2<br/>static secrets"]
        TR["Transit engine<br/>encrypt / decrypt API"]
    end
    VCO["Vault Config Operator<br/>CRs in Git → reconcile"] -.->|declarative config| VAULT

    ESO["External Secrets Operator"]
    DBE --> ESO
    KV -->|refresh ≤1h| ESO
    ESO -->|K8s Secret| POD["app Pods (env)"]
    REL["Reloader"] -.->|Secret changed → rollout| POD
    TR -->|"API call per encrypt/decrypt<br/>(key never leaves Vault)"| APP["app Pod (SA auth)"]

    GIT["Git (SealedSecret YAML)"] --> SSC["sealed-secrets controller"]
    SSC -->|K8s Secret| BOOT["bootstrap layer:<br/>Vault KMS cred / cert-manager DNS-01<br/>Longhorn R2"]

    classDef default fill:#26221D,stroke:#4A4232,color:#FCFAF6
    classDef vault fill:#075985,stroke:#38BDF8,color:#FCFAF6
    classDef bridge fill:#2F6B45,stroke:#7AC99A,color:#FCFAF6
    classDef boot fill:#8A4B12,stroke:#FDBA74,color:#FCFAF6
    class DBE,KV,TR vault
    class ESO,REL,SSC bridge
    class GIT,BOOT boot
Method Use for Rotation
Vault dynamic (Database engine) Postgres app creds where a restart / credential reload is acceptable TTL 24h, generated per lease
Vault static (KV via ESO) API keys, admin passwords, tokens — low churn but rotatable write new value to Vault → delivered within 1h
Vault Transit PII columns in app DBs (envelope encryption; only ciphertext touches the DB) manual key rotate + rewrap
SealedSecret Anything below Vault's dependency line — Vault's own KMS unseal cred, cert-manager DNS-01 IAM, Longhorn R2 token manual, low frequency

Current inventory (2026-07): the dynamic and Transit rails are deployed and were battle-tested by the previous kensan application, but their last consumers left with the kensan-legacy retirement (ADR-017) and the Keycloak revert (ADR-019) — zero active instances today; every live secret rides Vault static (ESO) or SealedSecret. Adding a consumer back is one platform-values/ file (engine README).

Design rationale

Three principles thread the whole design:

  1. Right-size, don't centralize. Moving a secret to Vault dynamic buys blast-radius reduction but costs a restart on every rotation — so IdPs and long-lived schema-owning services (Keycloak, Backstage) deliberately stay on Vault static. The same restraint applies to certificates: Vault PKI is intentionally not enabled while cert-manager + Let's Encrypt already finish the job (ADR-007).
  2. Declarative by default, scripts as a bounded exception. Vault mounts, policies, and auth roles are Vault Config Operator CRs living in Git — GitOps all the way into Vault, no terraform apply, no state file full of secrets. Only two operations stay script-managed (Transit key creation, OIDC config full-write), and that exception list is exhaustive by contract (ADR-015).
  3. The bootstrap layer must not depend on Vault. Whatever Vault needs to start (KMS auto-unseal cred), and whatever must exist before Vault is healthy (cert-manager's DNS-01 credential, Longhorn's backup target), is sealed into Git and decrypted in-cluster — breaking the circular dependency by construction. (Cilium's internal PKI, once on this list, moved to a cert-manager Certificate with auto-renewal.)

Concrete choices:

  • Rotation is closed end-to-end. ESO refreshes the K8s Secret; Reloader sees the change and rolls the workload. A rotation in Vault reaches running pods with no human in the loop.
  • Keycloak DB credentials are the stress test. First kept static (restart breaks SSO sessions — ADR-008), then moved to Vault dynamic with explicit cascade-risk mitigations (ADR-013), then reverted to static after operating it confirmed ADR-008's prediction — rotation restarts kept breaking SSO sessions (ADR-019). A worked example of the "right-size" principle being re-evaluated twice as operational evidence accumulated.
  • Transit keeps keys out of pods and DBs. Apps authenticate with their ServiceAccount and call encrypt/decrypt; the key material never leaves Vault, so a DB dump or pod compromise yields only ciphertext.
  • Vault's image tag is explicitly pinned — an upgrade of the secret core should never ride in on a chart bump unnoticed (ADR-011).