Stage 1: Vault HA Bootstrap¶
Bring up the Vault HA cluster with AWS KMS auto-unseal. The only manual step needed before GitOps deployment is sealing the AWS KMS credentials into a SealedSecret.
Prerequisites¶
- An AWS account with KMS Encrypt/Decrypt permissions
kubesealCLI installed (assumes the Sealed Secrets controller is already deployed viakubernetes/secrets/sealed-secrets/)- A KMS key
alias/vault-unseal-kensanalready created in ap-northeast-1
Vault chart configuration (reference)¶
Key parameters already defined in kubernetes/secrets/vault/values.yaml:
| Item | Value | Notes |
|---|---|---|
| replicas | 3 | Spread across nodes via anti-affinity |
| storage | local-path (10Gi × 2 PVC) | Planned switch after Longhorn adoption (roadmap) |
| seal | AWS KMS auto-unseal | alias/vault-unseal-kensan (ap-northeast-1) |
| image tag | 2.0.0 (explicit pin) |
ADR-011 |
| Agent Injector | disabled | Design bridges to K8s Secrets via ESO instead |
| audit device | enabled separately via bootstrap Terraform | vault audit enable file path=/vault/audit/audit.log |
Manual setup steps¶
1. Create an AWS IAM user¶
Create an IAM user restricted to KMS Encrypt / Decrypt only, and issue an access key.
2. Generate the raw secret into temp/¶
cat <<EOF > temp/vault-aws-kms-credentials-raw.yaml
apiVersion: v1
kind: Secret
metadata:
name: vault-aws-kms-credentials
namespace: vault
type: Opaque
stringData:
aws_access_key_id: "<ACCESS_KEY>"
aws_secret_access_key: "<SECRET_KEY>"
EOF
temp/*-raw.yaml is already .gitignored. Don't commit it.
3. Seal it with kubeseal¶
kubeseal --format=yaml < temp/vault-aws-kms-credentials-raw.yaml \
> kubernetes/secrets/vault/resources/aws-kms-credentials-sealed.yaml
4. Commit¶
Commit only the sealed YAML. The raw file can be discarded or left in temp/ (it's git-ignored either way).
Post-deploy operations¶
Once the Vault pods are up, run the following via bootstrap Terraform (bootstrap/vault/):
vault operator initvault audit enable file path=/vault/audit/audit.log- Enable the auth methods (kubernetes / oidc)
- Create the root policy
Those steps, why a throwaway Terraform is needed at all (the chicken-and-egg problem between VCO and Vault's own auth methods), and the disaster-recovery path are written beside the Terraform itself and published here as Vault root of trust.
Troubleshooting¶
- Pod stuck Sealed, won't join → vault-raft-join.md
- Need to rotate the AWS KMS credentials → regenerate the SealedSecret, overwrite the file in
resources/, let Argo CD sync