ADR-003: ApplicationSet Migration Strategy -- Coexistence with App of Apps¶
Status¶
Accepted
Date¶
2026-02-20
Context¶
The platform's Argo CD management has operated on the App of Apps pattern. platform-root-app recursively scans the applications/ directory and deploys the Application CRs for each category.
Issues¶
As operations progressed, the following problems became apparent:
- Boilerplate duplication: Apps within the same category have similar structures, but settings are scattered across individual
app.yamlfiles - Configuration inconsistency: Presence of finalizer, retry, and ServerSideApply varies per app
- Example: grafana has no SSA with retry, loki has SSA with retry, env-kensan-* has no finalizer
- Effort for new additions: Adding a new component requires copying and editing an entire
app.yaml
Options Considered¶
| Option | Summary |
|---|---|
| A. Keep everything as App of Apps | Maintain current state. Simple but doesn't resolve boilerplate issues |
| B. Migrate everything to ApplicationSet | Maximum consistency, but templatePatch becomes complex for apps with unique settings |
| C. Choose the optimal pattern per category (mixed) | Uniform categories use ApplicationSet; categories with many unique settings use individual Applications |
Decision¶
Adopt Option C: A mixed configuration selecting the optimal pattern per category.
Selection Criteria¶
(Drafted retroactively on 2026-06-07 by working backwards from the classification table below; review welcome.)
A category moves to ApplicationSet when all of the following hold; otherwise it stays on individual Applications:
- Uniform source shape — every app in the category has the same source structure (e.g. Helm chart + values ref), so one template covers all of them without per-app patches
- Additions are expected — new components arrive often enough that "drop in a
config.json" scaffolding pays for the generator's indirection - No dense per-app uniqueness — no sync-wave ordering, per-app
ignoreDifferences, or mixed tooling (Helm/Kustomize) that would force complextemplatePatchlogic - No external writer — nothing else (e.g. Backstage auto-commit) writes Application CRs into the category; generated and hand-written CRs in one directory would be confusing
Classification Results¶
| Category | App Count | Pattern | Primary Reason |
|---|---|---|---|
| Observability | 5 | ApplicationSet | Uniform structure with Helm multi-source, likely more additions in the future |
| Environments | 5 | ApplicationSet | Uniform structure with single source, frequent namespace additions |
| Network | 5 | Individual Application | sync-wave dependencies, unique ignoreDifferences |
| Security | 3 | Individual Application | Mixed Helm/Kustomize, sync-waves present |
| GitOps | 1 | Individual Application | Self-managed, complex ignoreDifferences |
| Backstage | 1 | Individual Application | Single Kustomize app |
| Apps | 3 | Individual Application | Auto-committed by Backstage, individual image tag updates |
ApplicationSet Design¶
Observability ApplicationSet¶
- Generator: Git File Generator (
kubernetes/observability/*/config.json) - Template: Helm multi-source (chart source + values ref)
- templatePatch: Add 3rd source (resources/) when
hasResources: "true" - Unified Policy: Apply finalizer, ServerSideApply, and retry to all apps
Environments ApplicationSet¶
- Generator: Git File Generator (
kubernetes/environments/*/config.json) - Template: Single source (Git directory)
- Unified Policy: Apply finalizer to all apps
Migration Safety Design¶
Zero-downtime migration achieved through a 2-commit strategy:
- Commit 1: Remove
resources-finalizer.argocd.argoproj.iofrom target Application CRs- After Argo CD sync, the finalizer is removed (no impact on managed resources)
- Commit 2: Add ApplicationSet CR and delete the old Application CRs
- root-app prunes the old Applications (no finalizer = no cascade deletion)
- ApplicationSet controller regenerates Application CRs with the same names
- Cluster workloads continue without interruption
Consequences¶
Benefits¶
- Centralized configuration management: Templates automatically unify finalizer / SSA / retry settings
- Simplified new additions: Just place
config.json+values.yamland an Application is auto-generated - Preservation of unique settings: Keeping individual Applications preserves sync-wave and ignoreDifferences complexity
Drawbacks¶
- Two mental models needed: "Where do I look for this app?" differs by category
- Mitigation: Place a pattern usage table in
applications/README.md - root-app deploys both ApplicationSets and Applications: A recognized pattern in the Argo CD community, but not a purely single pattern approach
Future Considerations¶
- If the Network category grows, evaluate whether ApplicationSet + merge generator can absorb sync-wave requirements
- If the Application CR format generated by Backstage templates becomes unified, consider converting the Apps category to ApplicationSet as well
Addendum (2026-06-07)¶
The classification table above no longer matches the implementation on two rows:
- Environments was ultimately implemented as individual Applications, not an ApplicationSet. After
dev/prod separation was abolished, the category's contents shifted to per-namespace lifecycle apps with
heterogeneous shapes (dedicated ns manifests,
directory.includefilters, whole-directory sources), which fails criterion 1 (uniform source shape). Noconfig.jsonfiles exist underkubernetes/environments/. - Two ApplicationSet-based capabilities were added later that this table predates:
vault-database-engine and vault-transit-engine (self-made chart + per-instance
platform-values/).
Observability remains the reference ApplicationSet implementation as designed. The current pattern usage is
documented in kubernetes/argocd/applications/README.md; this ADR's criteria (above) still describe the
decision logic accurately.