ADR-020: Namespace lifecycle app cleanup + backstage/app directory flatten¶
Status¶
Accepted (2026-07-12). Decided in conversation while auditing why kube-system and
kubernetes/namespaces/ felt structurally out of place compared to other components; the
review then extended to a related directory-nesting oddity in backstage/.
Part 1 executed (2026-07-12, #438 + follow-up): Phase 1 (duplicate resources/namespace.yaml
+ CreateNamespace=true) merged and synced; live cluster showed SharedResourceWarning as
expected (ArgoCD deliberately withholds automated sync/selfHeal on a resource declared by two
Applications, to avoid the two apps fighting over ownership in a loop). Ownership was
transferred with a one-time manual kubectl annotate namespace reloader
argocd.argoproj.io/tracking-id=reloader:/Namespace:reloader/reloader --overwrite, confirmed
reloader app went Synced/Healthy and the Reloader controller pod was undisturbed
throughout. Phase 2 (delete reloader-namespace app + duplicate namespace.yaml, #441)
completed the migration cleanly.
Part 2 executed (2026-07-13): backstage/app/* flattened to backstage/*. Beyond the
mechanical Makefile path fixes anticipated in the plan below, actually reading the files
surfaced one more real path dependency the plan hadn't named: packages/backend/Dockerfile's
COPY instructions were all prefixed app/..., because the Docker build context was
backstage/ (one cd .. from the old backstage/app/) with the Dockerfile itself living
inside that context at app/packages/backend/Dockerfile. Post-flatten the context is
backstage/ directly with no cd .. needed, so every COPY source lost its app/ prefix.
Verified by checking each COPY source path exists relative to the new backstage/ root
(all present except the two dist/*.tar.gz build outputs, which are expected to not exist
pre-build).
The app-config.yaml catalog target: paths for domains/*.yaml and organizations/*.yaml
needed no changes — they're documented as relative to the backend's own runtime execution
directory (packages/backend/dist/packages/backend), an internal detail of the app's package
layout, not of where backstage/ sits in the outer repo. Moving the app root doesn't change
that internal relative structure. This was confirmed empirically (see below), not just by
inspection.
Local dev-server verification was completed, after installing Node 22 via mise (host
default was v25, which fails to build the native isolated-vm module — engines pins
20 || 22; this mismatch was pre-existing and unrelated to the flatten, reproducible
identically on the pre-flatten layout). Running make dev under Node 22 surfaced one genuine,
pre-existing bug, unrelated to the flatten but only caught because the verification was
actually run: the fastapi-template catalog location used target: ../../templates/fastapi-template/template.yaml
(2 levels up) while the other two type: file locations in the same file correctly use 5
levels up, with a comment explaining why (packages/backend/dist/packages/backend is 5 levels
below the app root). All three locations resolve from the identical execution directory, so 2
levels was simply wrong — Backstage logged file .../packages/backend/dist/templates/fastapi-template/template.yaml
does not exist and silently skipped the template on every run, including before this change.
Fixed to ../../../../../templates/fastapi-template/template.yaml (5 levels, matching the
other two) as part of this PR. Re-running make dev after the fix showed zero
does not exist warnings and normal catalog/search collation.
The Node v25/v22 mismatch that blocked the first verification attempt exposed a real gap:
package.json declares engines: "20 || 22" and CI pins node-version: 22
(.github/workflows/app-ci.yml), but no local pin ever existed — no .nvmrc, .tool-versions,
or .mise.toml anywhere in git history. Only Yarn was actually pinned (bundled binary at
.yarn/releases/, packageManager: yarn@4.4.1). Added backstage/.tool-versions (node 22,
matching CI) so mise install picks the right version automatically going forward.
2026-07-12
Context¶
Part 1: namespace lifecycle apps¶
kubernetes/namespaces/README.md claims namespace management "splits into 2 types" but only
ever documented one (the component-colocated namespace.yaml pattern). The directory itself
holds nothing but that README — no manifest has ever lived there. Auditing the real
kubernetes/argocd/applications/namespaces/ app group (5 entries: kube-system, kyverno,
monitoring, reloader, sealed-secrets) surfaced that they fall into three genuinely
different categories, only one of which the README described:
| Category | Why it needs a dedicated namespaces/<name>/ app |
Examples |
|---|---|---|
| No owning component | The namespace has zero component manifests in this repo — nothing to attach resources/namespace.yaml + CreateNamespace=true to. kube-system is adopted from outside GitOps (created by kubeadm), not created by us. |
kube-system |
| Shared across sibling components | Multiple independent component dirs share one namespace, so no single owning Application makes sense (policy/kyverno + policy/kyverno-policies share policy/namespace.yaml; all 6 observability/* components share observability/namespace.yaml). |
kyverno, monitoring |
| Deliberate lifecycle decoupling | Single owner exists, but the ns is intentionally split out for protection — e.g. sealed-secrets: no finalizer + Prune=false so deleting/recreating the controller app can never delete the ns holding the sealing key. |
sealed-secrets |
reloader doesn't fit any of these three. secrets/reloader/ has exactly one owning
component (the reloader chart itself), no sibling shares its namespace, and its
reloader-namespace Application has ordinary finalizers + prune: true — no protective
config distinguishing it from a normal owned resource. It was split out as a copy of the
sealed-secrets/kyverno pattern without the underlying reason applying.
Meanwhile, components with a single owner and no adoption/sharing need (cert-manager,
external-secrets) already use the clean, existing Pattern A convention: resources/namespace.yaml
as an extra Application source alongside the Helm chart, with CreateNamespace=true — no
separate app at all. reloader should have followed this from the start.
Part 2: backstage/app/ nesting¶
Per ADR-018, Backstage's deploy manifests already
moved from backstage/manifests/ to kubernetes/backstage/, leaving backstage/ with a
single child: backstage/app/. The app/ nesting level existed to disambiguate the source
tree from the sibling manifests/ dir — that reason no longer applies, since manifests/ is
gone. apps/kensan/ (the other in-repo workload) has no such extra nesting; its source sits
directly at the top level. backstage/app/* is now a vestigial extra level for the same
reason ADR-018 already fixed once on the manifests side.
Auditing what depends on the current depth (backstage/app/ = 2 levels below repo root)
found:
- Doc-only, no functional risk:
CLAUDE.md,.claude/rules/environment-separation.md,kubernetes/README.md,docs/adr/018-*.md,docs/refactoring_master_plan.md,docs/secret-management/index.md,docs/bootstrapping/add-worker-node-m4neo.md,docs/getting-started/installation.md— all just mention the path in prose. - Mechanical, one
../fewer:backstage/app/Makefile—-include ../.env→-include .env;MANIFEST := ../../kubernetes/backstage/backstage-deployment.yaml→../kubernetes/backstage/backstage-deployment.yaml. - Not safely computable by inspection:
backstage/app/app-config.yamlhas threetype: filecatalogtarget:entries, two of which carry this comment:
# So we need to go up 5 levels: packages/backend/dist/packages/backend -> backstage-app/
target: ../../../../../catalog/domains/*.yaml
These resolve relative to the compiled backend's dist output layout, not to the
config file's on-disk location — Backstage's own path-resolution quirk. Removing one
nesting level does not necessarily mean removing exactly one ../; this can only be
confirmed by actually starting the backend and checking that the domains,
organizations, and fastapi-template catalog entries still load.
Decision¶
Part 1 — namespace lifecycle apps¶
-
Delete
kubernetes/namespaces/(the file-manifest directory — just the README, no real content). Replace it with the three-category table above, added tokubernetes/README.mdnext to the existing Pattern A/B section.kubernetes/argocd/applications/namespaces/(the ArgoCD Application definitions) is not touched — those apps stay; only the half-documented top-level doc-only directory goes away. -
Keep
kubernetes/kube-system/at top level, unchanged. It's not floating — it's the "no owning component" category, same structural justificationbackstage/has for being at top level per ADR-018 (a legitimate platform-level concern, not required to nest under a component-shaped directory). -
Merge
reloadernamespace ownership intosecrets/reloader's own Application, matching thecert-manager/external-secretspattern — executed as two phases, two PRs, because a plaingit mvis unsafe here:Why not a plain move:
reloader-namespace's source ispath: kubernetes/secrets/reloader,directory: {recurse: false, include: 'namespace.yaml'}. Moving the file out from under that path (even in the same commit that adds it elsewhere) makes that app's desired resource set empty immediately on merge. Withautomated: {prune: true}and aresources-finalizer.argocd.argoproj.iofinalizer, ArgoCD would prune the livereloaderNamespace as soon as it reconciles — taking the Reloader controller down — regardless of whethersecrets/reloader's new source has already taken over tracking. There's no git-commit-ordering trick that avoids this within a single merge, since both Applications reconcile independently and the prune isn't gated on the other Application's state.Phase 1 (this PR) — duplicate, don't move:
- Add
kubernetes/secrets/reloader/resources/namespace.yamlas a copy (same content) of the existingkubernetes/secrets/reloader/namespace.yaml. The original stays in place, untouched, marked with a comment explaining it's a deliberate temporary duplicate. - Add a third source to
kubernetes/argocd/applications/secrets/reloader/app.yamlpointing atkubernetes/secrets/reloader/resources, and flipCreateNamespace=false→CreateNamespace=true. reloader-namespace's source is completely untouched by this PR — it still finds its file at the original path, so it keeps syncing normally with zero risk of pruning. Both apps briefly co-own the same Namespace object (each sync flips the ArgoCD tracking annotation to itself) — expected, harmless flapping for the transfer window, not a steady state.- After merging, confirm in the live cluster that
secrets/reloader's Application isSynced/Healthyand applying the Namespace without errors.
Phase 2 (follow-up PR, after Phase 1 is confirmed live):
- Delete
kubernetes/argocd/applications/namespaces/reloader/app.yaml(and the now-empty dir) and the duplicatekubernetes/secrets/reloader/namespace.yamlin the same commit. Withreloader-namespacegone, nothing reads the old path anymore, so removing both together is safe and leaves no dangling duplicate.
- Add
Part 2 — backstage/app flatten¶
Flatten backstage/app/* up to backstage/* (source code becomes the direct content of
backstage/, matching the apps/kensan/ shape):
backstage/
├── packages/
├── templates/
├── catalog/ # if present under app/
├── Makefile
└── ... # everything currently under backstage/app/
git mv backstage/app/* backstage/app/.[^.]* backstage/(handle dotfiles separately:.yarn,.yarnrc.yml,.dockerignore,.eslintrc.js,.gitignore,.claude/, etc.), then remove the now-emptybackstage/app/.- Fix the mechanical one-level-shorter paths in the Makefile (
.env,MANIFEST). - Update
app-config.yaml's threetarget:entries empirically: start the backend locally after the move and confirmdomains,organizations, andfastapi-templatecatalog entries still resolve, adjusting../counts as needed rather than computing them by inspection. - Update the doc-only mentions listed above for accuracy.
Execution split: Part 1 and Part 2 ship as separate PRs sharing this one ADR, because
they need different verification methods — Part 1 is verified via ArgoCD/kubectl diff
against the live cluster, Part 2 is verified by actually running the Backstage backend
locally and checking catalog entities load. Bundling them would make one PR's review depend
on a verification step irrelevant to its own changes.
Alternatives considered¶
- Move
kube-systemintonamespaces/kube-system/(the original idea) — rejected once ADR-018's precedent made clear top-level placement doesn't require a component; moving it would just be churn with no structural gain. - Leave
reloadersplit as-is — zero risk, but keeps an unjustified inconsistency that looks like it protects something (mirroringsealed-secrets) when it doesn't. - Leave
backstage/app/as-is — zero risk, but keeps a nesting level whose only reason (disambiguating frommanifests/) was already removed by ADR-018.
Consequences¶
kubernetes/top level no longer has a doc-only directory whose own description contradicts its contents.- The three real reasons a namespace needs a dedicated lifecycle app are documented where
someone will actually find them (
kubernetes/README.md), instead of a half-written note. - One fewer ArgoCD Application (
reloader-namespaceremoved);reloader's ns now follows the same pattern ascert-manager/external-secrets. backstage/mirrorsapps/kensan/'s shape (source directly at the top level); one-time review + local-verification cost for the path changes inMakefileandapp-config.yaml.- Execution is a multi-step sequence (ownership transfer before cleanup in Part 1; move-then- verify in Part 2) — not single commits, for the reasons described above.