Skip to content

ADR-003: ApplicationSet Migration Strategy -- Coexistence with App of Apps

Status

Accepted

Date

2026-02-20

Context

The platform's Argo CD management has operated on the App of Apps pattern. platform-root-app recursively scans the applications/ directory and deploys the Application CRs for each category.

Issues

As operations progressed, the following problems became apparent:

  1. Boilerplate duplication: Apps within the same category have similar structures, but settings are scattered across individual app.yaml files
  2. Configuration inconsistency: Presence of finalizer, retry, and ServerSideApply varies per app
    • Example: grafana has no SSA with retry, loki has SSA with retry, env-kensan-* has no finalizer
  3. Effort for new additions: Adding a new component requires copying and editing an entire app.yaml

Options Considered

Option Summary
A. Keep everything as App of Apps Maintain current state. Simple but doesn't resolve boilerplate issues
B. Migrate everything to ApplicationSet Maximum consistency, but templatePatch becomes complex for apps with unique settings
C. Choose the optimal pattern per category (mixed) Uniform categories use ApplicationSet; categories with many unique settings use individual Applications

Decision

Adopt Option C: A mixed configuration selecting the optimal pattern per category.

Selection Criteria

(Drafted retroactively on 2026-06-07 by working backwards from the classification table below; review welcome.)

A category moves to ApplicationSet when all of the following hold; otherwise it stays on individual Applications:

  1. Uniform source shape — every app in the category has the same source structure (e.g. Helm chart + values ref), so one template covers all of them without per-app patches
  2. Additions are expected — new components arrive often enough that "drop in a config.json" scaffolding pays for the generator's indirection
  3. No dense per-app uniqueness — no sync-wave ordering, per-app ignoreDifferences, or mixed tooling (Helm/Kustomize) that would force complex templatePatch logic
  4. No external writer — nothing else (e.g. Backstage auto-commit) writes Application CRs into the category; generated and hand-written CRs in one directory would be confusing

Classification Results

Category App Count Pattern Primary Reason
Observability 5 ApplicationSet Uniform structure with Helm multi-source, likely more additions in the future
Environments 5 ApplicationSet Uniform structure with single source, frequent namespace additions
Network 5 Individual Application sync-wave dependencies, unique ignoreDifferences
Security 3 Individual Application Mixed Helm/Kustomize, sync-waves present
GitOps 1 Individual Application Self-managed, complex ignoreDifferences
Backstage 1 Individual Application Single Kustomize app
Apps 3 Individual Application Auto-committed by Backstage, individual image tag updates

ApplicationSet Design

Observability ApplicationSet

  • Generator: Git File Generator (kubernetes/observability/*/config.json)
  • Template: Helm multi-source (chart source + values ref)
  • templatePatch: Add 3rd source (resources/) when hasResources: "true"
  • Unified Policy: Apply finalizer, ServerSideApply, and retry to all apps

Environments ApplicationSet

  • Generator: Git File Generator (kubernetes/environments/*/config.json)
  • Template: Single source (Git directory)
  • Unified Policy: Apply finalizer to all apps

Migration Safety Design

Zero-downtime migration achieved through a 2-commit strategy:

  1. Commit 1: Remove resources-finalizer.argocd.argoproj.io from target Application CRs
    • After Argo CD sync, the finalizer is removed (no impact on managed resources)
  2. Commit 2: Add ApplicationSet CR and delete the old Application CRs
    • root-app prunes the old Applications (no finalizer = no cascade deletion)
    • ApplicationSet controller regenerates Application CRs with the same names
    • Cluster workloads continue without interruption

Consequences

Benefits

  • Centralized configuration management: Templates automatically unify finalizer / SSA / retry settings
  • Simplified new additions: Just place config.json + values.yaml and an Application is auto-generated
  • Preservation of unique settings: Keeping individual Applications preserves sync-wave and ignoreDifferences complexity

Drawbacks

  • Two mental models needed: "Where do I look for this app?" differs by category
  • Mitigation: Place a pattern usage table in applications/README.md
  • root-app deploys both ApplicationSets and Applications: A recognized pattern in the Argo CD community, but not a purely single pattern approach

Future Considerations

  • If the Network category grows, evaluate whether ApplicationSet + merge generator can absorb sync-wave requirements
  • If the Application CR format generated by Backstage templates becomes unified, consider converting the Apps category to ApplicationSet as well

Addendum (2026-06-07)

The classification table above no longer matches the implementation on two rows:

  • Environments was ultimately implemented as individual Applications, not an ApplicationSet. After dev/prod separation was abolished, the category's contents shifted to per-namespace lifecycle apps with heterogeneous shapes (dedicated ns manifests, directory.include filters, whole-directory sources), which fails criterion 1 (uniform source shape). No config.json files exist under kubernetes/environments/.
  • Two ApplicationSet-based capabilities were added later that this table predates: vault-database-engine and vault-transit-engine (self-made chart + per-instance platform-values/).

Observability remains the reference ApplicationSet implementation as designed. The current pattern usage is documented in kubernetes/argocd/applications/README.md; this ADR's criteria (above) still describe the decision logic accurately.