Vault storage migration: local-path → Longhorn¶
Status: executed (2026-07-06). Every phase below was carried out exactly as written and completed with zero downtime and quorum maintained throughout (the leader stepped down and moved to vault-2). Kept from here on as a reference procedure for rebuilding storage on any Vault node in general.
Background¶
PR #324 (disaster recovery) changed the storageClass in
kubernetes/secrets/vault/values.yaml to longhorn, but the live StatefulSet
kept running on local-path. Since volumeClaimTemplates is immutable, Argo CD's
server-side diff fails permanently, and the vault Application shows
SYNC STATUS: Unknown (ComparisonError) indefinitely.
StatefulSet.apps "vault" is invalid: spec: Forbidden: updates to statefulset spec for fields
other than 'replicas', 'ordinals', 'template', ... are forbidden
On top of that, local-path is node-local (a Pod can't move to a different node) and
uses reclaim=Delete (deleting the PVC instantly destroys the physical data) —
an unsuitable place to keep Vault's raft data.
Current state (confirmed live, 2026-07-06)¶
| Item | Value |
|---|---|
| StatefulSet | vault namespace, replicas=3, live VCT: local-path 10Gi / git: longhorn |
| PVC | data-vault-{0,1,2} + audit-vault-{0,1,2} (on worker2 / m4neo / worker1) |
| Unseal | AWS KMS auto-unseal (alias/vault-unseal-kensan) — no manual unseal needed |
| Raft join | retry_join ×3 already configured (docs/runbooks/vault-raft-join.md) — no manual join needed |
Thanks to these two things (auto-unseal + retry_join), a member whose data was wiped recovers automatically just by recreating its Pod — it unseals itself, rejoins the raft cluster, and syncs a snapshot from the leader. That makes a per-node "strip it and rejoin" strategy viable.
Strategy¶
- Phase A: delete the StatefulSet with
--cascade=orphan→ Argo CD recreates the StatefulSet using the longhorn VCT (the Pods stay running throughout). → clears the ComparisonError,vaultapp returns to Synced - Phase B: migrate members one at a time (always keeping quorum at 2/3): raft remove-peer → delete PVC/Pod → the new Pod comes up on a longhorn PVC and auto-rejoins
- Phase C: verification and cleanup
Pre-flight checks (all must pass before starting)¶
# 1. raft is healthy with 3 peers (1 leader + 2 followers)
kubectl exec -n vault vault-0 -c vault -- sh -c 'VAULT_TOKEN=$(cat /tmp/token 2>/dev/null || echo $VAULT_TOKEN) vault operator raft list-peers'
# NOTE: requires login first. Run `vault login` with an OIDC admin or the root token before this
# 2. Take a raft snapshot and stash it locally (a last-resort safety net)
kubectl exec -n vault vault-0 -c vault -- vault operator raft snapshot save /tmp/pre-migration.snap
kubectl cp vault/vault-0:/tmp/pre-migration.snap ./temp/vault-pre-migration-$(date +%Y%m%d).snap -c vault
# 3. Confirm KMS auto-unseal is working (e.g. it went Sealed:false after a recent Pod restart)
kubectl exec -n vault vault-0 -c vault -- vault status | grep -E "Sealed|Total Shares" # Sealed: false / Recovery Seed
# 4. Longhorn is healthy (every volume healthy, >30Gi free)
kubectl get volumes.longhorn.io -n longhorn-system --no-headers | awk '$3!="attached" && $2!="detached"' | head
# 5. ESO / dynamic secrets are healthy (a baseline check so a break during migration is noticeable)
kubectl get externalsecret -A --no-headers | grep -cv SecretSynced # → should be 0
Phase A: recreate the StatefulSet (no Pod downtime, clears ComparisonError)¶
# 1. Delete only the StatefulSet (Pods and PVCs stay)
kubectl delete sts vault -n vault --cascade=orphan
# 2. Wait for Argo CD to recreate the StatefulSet from Git (the longhorn VCT)
kubectl get application vault -n argocd -w # → should become Synced / Healthy
kubectl get sts vault -n vault -o jsonpath='{.spec.volumeClaimTemplates[0].spec.storageClassName}' # → longhorn
# 3. Confirm the Pods weren't picked up and restarted (AGE is unchanged)
kubectl get pods -n vault
Check: the vault app's SYNC STATUS flips from Unknown to Synced.
Existing Pods keep running on their old local-path PVCs (the VCT only takes effect for new Pods).
Phase B: migrate members one at a time (vault-2 → vault-1 → vault-0)¶
Save the leader for last. Check the active node with vault status | grep "HA Mode" —
if vault-2 or vault-1 is active, run vault operator step-down on it first to move leadership away.
For each member (using vault-2 as the example):
# 1. Cleanly remove the peer from raft
kubectl exec -n vault vault-0 -c vault -- vault operator raft remove-peer vault-2
# 2. Schedule the PVCs for deletion (they're in use, so this waits in Terminating)
kubectl delete pvc data-vault-2 audit-vault-2 -n vault --wait=false
# 3. Delete the Pod → the PVC deletion completes, and the StatefulSet creates a new Pod + longhorn PVCs
kubectl delete pod vault-2 -n vault
# 4. Confirm the new Pod comes up and recovers automatically (fully automatic via KMS auto-unseal + retry_join)
kubectl get pvc -n vault | grep vault-2 # → should now be longhorn
kubectl get pod vault-2 -n vault -w # → should reach 2/2 Running
kubectl exec -n vault vault-2 -c vault -- vault status | grep Sealed # → should be false
# 5. Confirm all 3 raft peers are present and synced before moving to the next member
kubectl exec -n vault vault-0 -c vault -- vault operator raft list-peers
kubectl exec -n vault vault-0 -c vault -- vault operator raft autopilot state | grep -A5 Servers # healthy: true
Never do this: work on 2 members at the same time (loses quorum = total outage). Don't move to the next member until the current one's re-sync is fully complete (autopilot healthy).
When it's vault-0's turn, if it's the active leader, step it down first:
kubectl exec -n vault vault-0 -c vault -- vault operator step-down
# Confirm the active leader moved to vault-1/2, then apply the same procedure to vault-0
# Run remove-peer against the new leader:
kubectl exec -n vault vault-1 -c vault -- vault operator raft remove-peer vault-0
Phase C: verification and cleanup¶
# All PVCs are longhorn
kubectl get pvc -n vault # all 6 should show STORAGECLASS=longhorn
# raft has 3 peers / autopilot healthy / app Synced
kubectl exec -n vault vault-0 -c vault -- vault operator raft list-peers
kubectl get application vault -n argocd
# Consumers are unaffected (ESO sync, Keycloak dynamic creds, cert-manager)
kubectl get externalsecret -A --no-headers | grep -cv SecretSynced # → should be 0
# Longhorn side: confirm all 6 vault volumes are healthy and picked up by the RecurringJob (R2 backup)
kubectl get volumes.longhorn.io -n longhorn-system | grep -c healthy
- Keep the pre-migration raft snapshot around for about a week, then discard it
- local-path PVs use reclaim=Delete, so they vanish automatically when their PVC is
deleted (visually confirm no leftovers remain under
/opt/local-path-provisioner/on the node)
Rollback¶
- If Phase A fails (the StatefulSet recreation ends up in a bad state):
kubectl delete sts vault --cascade=orphan→ the old-VCT StatefulSet can be hand-applied again even without a git revert (though the default plan is to always move forward — the Pods stay alive throughout, so there's no rush) - If a member fails to recover in Phase B: quorum is still alive on the remaining 2
peers. Try the manual join procedure in
vault-raft-join.md→ if that still doesn't work, recreate the PVC/Pod once more. As a last resort, restore from the pre-migration snapshot viavault operator raft snapshot restore - Never do this: start on the next member while the current one is still in a failed-recovery state
Remaining work (out of scope for this runbook)¶
- Migrating the 5 local-path PVCs in the
monitoringnamespace (prometheus / loki / tempo / grafana / alertmanager) — since this is retention data, "recreate the StatefulSet and accept data loss" is also a viable option - Once every local-path PVC is gone: remove
local-path-provisionerandapplications/namespaces/local-path-storage(the W15 completion condition)