# kensan-lab > Enterprise-grade Kubernetes on bare-metal — a reference architecture for platform engineering. Enterprise-grade Kubernetes on bare-metal — a reference architecture for platform engineering. Argo CD GitOps, Istio service mesh, Cilium CNI, Vault, Keycloak, Backstage, and full observability (Prometheus, Grafana, Loki, Tempo), running on Raspberry Pis and a mini PC. # Overview # kensan-lab **Enterprise-grade Kubernetes on bare-metal — a reference architecture for platform engineering.** A bare-metal Kubernetes homelab built with technologies typical of enterprise platform engineering — Argo CD for GitOps, Istio for service mesh, Backstage for developer self-service, and observability with Prometheus, Grafana, Loki, and Tempo. All running on Raspberry Pis and a mini PC. [Try it locally](https://yu-min3.github.io/kensan-lab/getting-started/try-it-with-kind/index.md) [See the Showcase](https://yu-min3.github.io/kensan-lab/showcase/index.md) Try kind first; treat bare metal as a reference The kind path is the tested, automated way to experience the platform. The bare-metal pages document the live cluster's design and current build steps, but a clean-room bootstrap has not yet been verified end to end. Ansible and Makefile automation is planned. Start with the [kind walkthrough](https://yu-min3.github.io/kensan-lab/getting-started/try-it-with-kind/index.md); use the [bare-metal bootstrap pages](https://yu-min3.github.io/kensan-lab/bootstrapping/index.md) as a reference. ## Architecture Everything inside the cards runs in-cluster — click to enlarge Each domain has an architecture page — design thesis, component map, diagrams, and the rationale behind the choices: | Domain | One-liner | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | [Argo CD](https://yu-min3.github.io/kensan-lab/architecture/argocd/index.md) | One root App-of-Apps, Git as the only actor; ApplicationSet only where structure is uniform | | [Network](https://yu-min3.github.io/kensan-lab/architecture/network/index.md) · [Cloudflare Tunnel](https://yu-min3.github.io/kensan-lab/architecture/cloudflare-tunnel/index.md) | Cilium CNI + L2 LB, Istio mesh, Gateway API edge; Zero Trust internet exposure | | [Auth](https://yu-min3.github.io/kensan-lab/architecture/auth/index.md) | Keycloak OIDC, enforced once at the Gateway (oauth2-proxy ext_authz) | | [Secrets](https://yu-min3.github.io/kensan-lab/architecture/secrets/index.md) | Vault at the core, four delivery methods, SealedSecret bootstrap lane | | [Observability](https://yu-min3.github.io/kensan-lab/architecture/observability/index.md) | Three pillars through one OTel pipe; the monitoring is itself monitored | | [Storage](https://yu-min3.github.io/kensan-lab/architecture/storage/index.md) | Longhorn only; Retain + Prune=false + off-cluster R2 backups | | [Backstage](https://yu-min3.github.io/kensan-lab/architecture/backstage/index.md) | IDP front door — golden path templates, catalog, TechDocs | Cross-cutting overviews: [Infrastructure](https://yu-min3.github.io/kensan-lab/architecture/index.md) and [Cluster health monitoring](https://yu-min3.github.io/kensan-lab/architecture/cluster-health-monitoring/index.md). ## Showcase The repository describes a live platform. Start with the [Showcase](https://yu-min3.github.io/kensan-lab/showcase/index.md) to see the visual proof points: Argo CD app health, Grafana cluster health, and Hubble network visibility (Backstage and kensan views coming). ## How to use this site This site is the **single source of truth** for the current state of kensan-lab. Articles on Zenn / dev.to are point-in-time deep dives — they capture the journey but may go stale. Always cross-check against the docs here. | If you want to… | Start here | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Experience the platform locally | [Try it with kind](https://yu-min3.github.io/kensan-lab/getting-started/try-it-with-kind/index.md) | | Understand the platform | [Architecture](#architecture) above, then [ADRs](https://yu-min3.github.io/kensan-lab/adr/index.md) | | See the running system | [Showcase](https://yu-min3.github.io/kensan-lab/showcase/index.md) | | Study the bare-metal build | [Bare-metal reference](https://yu-min3.github.io/kensan-lab/bootstrapping/index.md) — not yet verified as a clean-room bootstrap | | Operate it day-to-day | [Secret Management](https://yu-min3.github.io/kensan-lab/secret-management/index.md), [Runbooks](https://yu-min3.github.io/kensan-lab/runbooks/index.md) | | Read the back-stories | [Articles](https://yu-min3.github.io/kensan-lab/articles/index.md) — Zenn / dev.to deep-dive index | | See what broke and why | [Incidents](https://yu-min3.github.io/kensan-lab/incidents/index.md) | | Feed this site to an LLM | [`llms.txt`](https://yu-min3.github.io/kensan-lab/llms.txt) (curated) / [`llms-full.txt`](https://yu-min3.github.io/kensan-lab/llms-full.txt) (full content) | ## Tech Stack & Hardware Tech stack table, hardware inventory, and feature highlights are in the [Top README](https://github.com/yu-min3/kensan-lab/blob/main/README.md) — the single source of truth. ## License [Apache-2.0](https://github.com/yu-min3/kensan-lab/blob/main/LICENSE) # Showcase This page is the visual entry point for the running kensan-lab platform. The repository is a reference architecture, but it is also a live system — these views are the proof. Six views tell the story: the physical bare-metal cluster, deployment health (Argo CD), observability (Grafana), the developer platform (Backstage), zero-trust network visibility (Hubble), and the app the platform exists to run. ## Hardware — bare metal, literally The platform behind the dashboards: three Raspberry Pi 5 nodes and one amd64 worker, connected through a managed switch. The architecture in this repository runs on this hardware. ## Argo CD — the cluster is GitOps-managed Every platform and app component reconciled by Argo CD — 38 Synced / 0 OutOfSync. A live GitOps system, not static manifests. ## Grafana — the operational layer Cluster Health: node liveness and resources across the Pi 5 nodes and the amd64 worker — including the CPU-temperature panel that hardware reality demands. ## Backstage — the Internal Developer Platform The kensan component in Backstage: ownership and system, a live Relations graph, and one click to its TechDocs — the catalog that turns "what runs here" into self-service. ## Hubble — zero-trust network visibility Cilium/Hubble makes every service-to-service flow observable — here app-kensan's real traffic: Syncthing sync in, telemetry out to the OTel Collector. ## kensan — the app that runs on the platform kensan is **not the point of this repository — the platform is.** It's a dogfooding workload: a real app that exercises the platform end-to-end (deployed by Argo CD, cataloged in Backstage, secured at the Gateway, observed like any other service), so the platform is proven against something real rather than a toy. kensan (Go single binary + Whetstone SPA), a file-based knowledge & goal manager that reads Markdown as its single source of truth. Shown with a **sample workspace** — fictional data, not a live view. The UI is the author's personal tool and is currently Japanese-only (see [demo-workspace](https://github.com/yu-min3/kensan-lab/blob/main/apps/kensan/demo-workspace/) to run it locally). **Capture guidelines (for contributors adding new screenshots)** - Use real running screens only. No mock dashboards or generated images. - Redact domains, user names, tokens, cluster IDs, and IP addresses that are not already public. - Prefer a 16:9 browser viewport around 1440px wide; save as PNG. - Keep captions outcome-focused: what the screen *proves*, not how to operate the tool (the captions above already follow this — adjust only if the shot differs). # Try it in one command # Try it in one command This is the shortest path through kensan-lab. In about ten minutes you will inspect a GitOps-managed cluster, open an SSO-protected demo, create a second application through Backstage, merge its pull request, watch its CPU usage change in Grafana, and scale it twice — once the way the platform accepts, once the way it undoes. Pull requests that touch the platform stand this same environment up on CI and sign in to it, so what follows is checked rather than merely written down. The bare-metal cluster is not required, and that is why this page exists. kensan-lab runs on hardware at home, and the [bootstrapping guide](https://yu-min3.github.io/kensan-lab/bootstrapping/index.md) documents that cluster as a reference — it describes what is running, not a clean-room install, and standing one up from nothing is not automated yet. So the way in is a disposable, single-node kind cluster that runs the same manifests and binds its gateway only to `127.0.0.1`. ## 1. Start the platform You need Docker with at least 8 GiB of memory, plus `kind`, `kubectl`, `helm`, `curl`, and `make`. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/), set **Settings → Resources → Memory** to at least 8 GiB, then install the command-line tools: ```bash brew install kind kubectl helm ``` Install [Docker Engine](https://docs.docker.com/engine/install/) and the [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation), [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/), and [Helm](https://helm.sh/docs/intro/install/) CLIs. Then clone and start the environment: ```bash git clone https://github.com/yu-min3/kensan-lab cd kensan-lab make try ``` `make try` checks the prerequisites, builds the demo and Backstage images from your checkout, creates the cluster, and waits until every Argo CD Application is healthy. The initial image build can take several minutes; later runs reuse Docker's layer cache. The account used across the platform is fixed for this disposable environment: | User | Password | Used for | | ------ | -------- | --------------------------------------------------------- | | `demo` | `demo` | Demo apps, Argo CD SSO, Backstage, Grafana SSO, and Gitea | ## 2. Accept the local certificate Every browser URL uses HTTPS. cert-manager creates a local certificate authority and signs `*.127-0-0-1.sslip.io`, but your browser does not trust that new CA. The **Your connection is not private** warning is therefore expected. For a quick walkthrough, choose **Advanced** and then **Proceed to …127-0-0-1.sslip.io (unsafe)**. The gateway listens only on localhost. If you prefer to trust the generated root explicitly, export it before opening the sites: ```bash kubectl -n cert-manager get secret explore-ca-tls \ -o jsonpath='{.data.ca\.crt}' | base64 --decode \ > /tmp/kensan-lab-explore-ca.crt ``` ```bash sudo security add-trusted-cert -d -r trustRoot \ -k /Library/Keychains/System.keychain \ /tmp/kensan-lab-explore-ca.crt ``` Remove it when the walkthrough is finished: ```bash sudo security delete-certificate -c "kensan-lab explore" \ /Library/Keychains/System.keychain ``` Open the browser's certificate settings, import `/tmp/kensan-lab-explore-ca.crt` under **Authorities**, and allow it to identify websites. Remove the `kensan-lab explore` authority afterwards. Importing the root is optional. The environment never changes the host trust store automatically. ## 3. See GitOps in Argo CD Open [Argo CD](https://argocd.127-0-0-1.sslip.io) and choose **Log in via Keycloak**. Sign in with `demo` / `demo`. Open `explore-root`. It is the root Application that creates the platform's other Applications. `app-demo`, `backstage`, `grafana`, and the rest should be `Synced` and `Healthy`. Argo CD is the GitOps controller for this cluster. It continuously traces the `kensan-lab` repository in the in-cluster Gitea, renders the declared Helm and Kubernetes resources, and reconciles them into the cluster. A green `Synced / Healthy` application therefore means both that the live resources match Git and that their workloads are running successfully. The same state is available from the terminal: ```bash make explore-status ``` ## 4. Open the existing demo Open the [demo application](https://demo.127-0-0-1.sslip.io). Keycloak should reuse the session from Argo CD, so no second password prompt appears. The app displays its name, theme, greeting, and the identity headers attached by the gateway. The application itself contains no login implementation: Istio asks oauth2-proxy to authenticate the request before it reaches the pod. The load button near the bottom belongs to step 7; leave it alone for now. ``` sequenceDiagram participant Browser participant Gateway as Istio Gateway participant Auth as oauth2-proxy + Keycloak participant App as demo app Browser->>Gateway: HTTPS request Gateway->>Auth: Is this session valid? Auth-->>Gateway: Yes + user identity Gateway->>App: Request + identity headers App-->>Browser: Render the demo ``` ## 5. Create a different app in Backstage Open [Backstage](https://backstage.127-0-0-1.sslip.io), then select **Create → Golden Path Application**. The Explore catalog has one owner, `demo-team`, so the walkthrough does not ask you to choose among production teams. **The application name is yours.** One field decides what the platform builds, so the rest of this page writes it as `` in the text and `$APP` in the commands. The screenshots were taken with `app2`. | Field | Value | | ---------------- | --------------------------------- | | Application Name | `` — anything you like | | Description | `Second walkthrough application` | | Repository | owner `demo`, repository `` | | Theme | `night` | | Greeting | `Hello from the golden path` | Everything the platform names after it: | Thing | Named | With `app2` | | ------------------------------ | --------------------------- | ------------------------- | | Argo CD Application, namespace | `app-` | `app-app2` | | Deployment, Service | `` | `app2` | | Hostname | `.127-0-0-1.sslip.io` | `app2.127-0-0-1.sslip.io` | | Gitea repository | `demo/` | `demo/app2` | Set the name once in the terminal you are going to use, and every command below works unchanged: ```bash APP=app2 # whatever you typed in the form ``` Review the values and press **Create**. Backstage now: ``` flowchart TD A[Backstage form] --> B[Create NAME repository in local Gitea] B --> F[Gitea Actions tests and builds NAME] F --> G[Push commit-SHA image] G --> H[Record image tag in deploy/values.yaml] H --> C[Open platform-config PR] C --> D[Register NAME in the catalog] C --> E[Register the new SSO callback] C --> I{You merge the PR} I --> J J[Argo CD discovers app-NAME] J --> K[NAME is running] ``` The task stays open while Gitea Actions tests the generated source, builds an image for that repository alone, pushes it, and writes its immutable commit SHA into `deploy/values.yaml`. A failed build fails the Backstage task and no platform pull request is created. A completed task means the pull request is safe to review and merge. ## 6. Merge the local pull request Follow **Review the kensan-lab pull request** from Backstage, or open [the `kensan-lab` pull requests in Gitea](https://gitea.127-0-0-1.sslip.io/demo/kensan-lab/pulls). Gitea does not share the Keycloak browser session, but it uses the same memorable credentials: sign in with `demo` / `demo`. You are the platform administrator for this part of the walkthrough. The new Platform Config PR should already be waiting in `demo/kensan-lab`. It adds two files under `environments/kind/generated-applications/app-/`; merge it. Merging does not deploy from Gitea directly. It updates the Git source of truth; Argo CD notices that change on its next poll, creates the child Application, and reconciles the workload. This can take about four minutes after the merge. ``` flowchart LR M[1. Merge PR
in Gitea] --> G[2. main changes
in demo/kensan-lab] G -->|Argo CD polls Git| A[3. Argo CD
creates app-NAME] A --> K[4. Kubernetes
deploys the SHA image] K --> B[5. Open
NAME.127-0-0-1.sslip.io] ``` Watch the new Argo CD Application appear: ```bash kubectl -n argocd get application app-$APP -w ``` When it is `Synced` and `Healthy`, open `https://.127-0-0-1.sslip.io` and compare it with the [original demo](https://demo.127-0-0-1.sslip.io). Your service runs an image built from its own repository, and its night theme and greeting came from Git-managed runtime values rather than from that image. Editing the service's own source and watching a new image reach the cluster is the natural next question. It costs a build, so it lives in [Source change exercise](https://yu-min3.github.io/kensan-lab/getting-started/kind-explained/#source-change-exercise) rather than here. ## 7. Watch CPU rise and fall in Grafana Open the [Explore App Runtime dashboard](https://grafana.127-0-0-1.sslip.io/d/explore-app-runtime/explore-app-runtime?refresh=10s) in Grafana. Choose **Sign in with Keycloak** if asked. It shows one Deployment's CPU, desired and available replicas, request rate, and request latency. It opens on the built-in `demo` application. **Set the Namespace picker to `app-` and the Workload picker to `` first.** Both pickers are filled from the cluster, so they list whatever you called your service — and the panels stay empty until they point at a workload that exists. Then open your application in another tab and press **Generate load for 90s**. It is the same button you saw on the demo app in step 4, and it belongs to the service you just deployed: it occupies one core inside the pod and counts down while it does. The line does not jump the moment you press: Grafana is reading the pod's real CPU through Prometheus, so it climbs over the next few samples, holds, and drops back once the ninety seconds are up. ## 8. Scale it, and watch Git win The Replicas panel is where the GitOps contract becomes visible. Change the replica count twice — once from the cluster, once from Git — and watch which one survives. **From the cluster.** Argo CD puts this back within a second or two: ```bash kubectl -n app-$APP scale deployment/$APP --replicas=3 kubectl -n app-$APP get deployment/$APP -w ``` Argo CD is faster than one scrape interval, so the Replicas panel usually never shows the 3 at all: the cluster was only briefly wrong. **From Git.** In your service's repository in Gitea: 1. open `deploy/values.yaml` 1. change `replicas: 1` to `replicas: 3` with the edit button 1. commit to `main` Nothing pushes this one back. Argo CD polls Git every three minutes and a change can miss one tick, so give it up to five — or press **Refresh** on the `app-` Application in Argo CD to skip the wait. Argo CD then shows three pods under the Deployment, and the Replicas panel in Grafana climbs to 3. ``` flowchart TD S["Deployment
1 replica"] S -->|"kubectl scale --replicas=3"| X["3 replicas,
for a second or two"] X -->|"Argo CD: Git still says 1"| S S -->|"commit replicas: 3
to deploy/values.yaml"| Y["Argo CD syncs
Application shows 3/3 Healthy"] Y --> Z["Deployment
3 replicas"] ``` No image is built for that one. The workflow ignores `deploy/`, because what lives there is read at runtime rather than baked into the image: | What you change | Image rebuilt | What you see | | ---------------------- | ------------- | ------------------------------------------ | | `frontend/src/App.tsx` | yes | New SHA tag, Argo CD replaces the pod | | `deploy/values.yaml` | no | Argo CD applies the value on the next sync | | `kubectl scale` | — | Undone; Git never said 3 | ## 9. Clean up If you imported the local CA, remove it from the trust store first. Then delete the entire disposable cluster: ```bash make explore-down ``` ## Want the details? Continue to [How the kind environment works](https://yu-min3.github.io/kensan-lab/getting-started/kind-explained/index.md) for the component map, per-application build path, Gateway API and certificate design, optional Kyverno/storage/CNI exercises, limitations, and troubleshooting. For the hardware side, the [bare-metal bootstrap guide](https://yu-min3.github.io/kensan-lab/bootstrapping/index.md) is the reference described above; Ansible and Makefile automation is planned. # How the kind environment works The Explore environment is a **tested slice of the platform with explicit substitutions**. It is built for learning and pull-request validation; it does not pretend that one laptop reproduces bare-metal networking, storage, or failure behavior. Follow the [walkthrough](https://yu-min3.github.io/kensan-lab/getting-started/try-it-with-kind/index.md) first. This page explains what it did and collects the optional technical exercises. ## Component map ``` flowchart TB Browser[Browser on localhost] --> Gateway[Istio Gateway
ports 80 + 443] Gateway --> OIDC[oauth2-proxy + Keycloak] Gateway --> Argo[Argo CD] Gateway --> Backstage[Backstage] Gateway --> Grafana[Grafana] Gateway --> Apps[demo + generated apps] Backstage --> Gitea[Gitea inside kind] Gitea --> Argo Argo --> Cluster[Kubernetes resources] Prom[Prometheus] --> Cluster Grafana --> Prom Policy[Kyverno] --> Cluster ``` | Component | Explore behavior | | ------------------------------------------------- | --------------------------------------------------------------------------------------------- | | [Argo CD](https://argocd.127-0-0-1.sslip.io) | Reconciles the app-of-apps tree from the in-cluster Gitea repository | | [Backstage](https://backstage.127-0-0-1.sslip.io) | Creates a repository, platform PR, catalog entity, and SSO callback without an external token | | [Gitea](https://gitea.127-0-0-1.sslip.io) | Receives the current checkout and holds all disposable repositories | | [Grafana](https://grafana.127-0-0-1.sslip.io) | Reads local Prometheus metrics from the cluster and applications | | [Keycloak](https://auth.127-0-0-1.sslip.io) | Provides one disposable `demo` / `demo` identity | | Istio + Gateway API | Routes browser services over HTTPS without port-forwarding | | cert-manager | Creates a local CA and wildcard certificate | | Kyverno | Runs the production policy set in Audit mode | | `longhorn` StorageClass | Uses kind's local-path provisioner under the production class name | ## How a generated app becomes an image Explore includes a disposable CI supply chain so that the code in each generated repository is the code that actually runs. ``` flowchart TD F[Backstage form] --> R[Generated Gitea repository] R --> T[Gitea Actions: test] T --> B[Build repository source] B --> I[Push commit-SHA image
to disposable registry] I --> V[Commit SHA tag to deploy/values.yaml] V --> P[Backstage opens platform PR] P --> M[Platform administrator merges PR] M --> A[Argo CD sync] A --> Pod[New application pod] ``` `make try` builds and loads the fixed `app-demo` image so the platform has something to show immediately. It also builds Backstage from the current checkout, then starts an Explore-only Gitea Actions runner and registry. Every repository created afterward runs tests, builds its own Python and React source, pushes an image tagged with the source commit SHA, and commits that tag to `deploy/values.yaml`. The tag commit contains `[skip ci]`, preventing an identical recursive build. The seeded copy of this repository is the one exception. Its workflows are written for GitHub's hosted runners, and Gitea reads `.github/workflows` too, so `make try` turns Actions off for that repository alone. Left on, the platform pull request you are asked to merge would arrive showing failed checks that nothing in this cluster could ever satisfy. Changing source on `main` repeats that path and changes the Deployment pod template, so Kubernetes replaces the pod. Changing `deploy/` does not: the workflow declares `paths-ignore: ["deploy/**"]`, because the theme, the greeting and the replica count are read at runtime or applied by Argo CD. Editing them is a sync, not a build. That file's own comment has always said so; the filter is what made it true. The runner uses privileged Docker-in-Docker. That is acceptable only because this cluster is single-user, bound to localhost and deleted as a unit. Bare metal uses GitHub Actions, GHCR and a multi-architecture build instead; the Explore image contains only the host architecture. ## Identity and request routing | Workload | Authentication path | Reason | | --------------------------- | ---------------------------------- | -------------------------------------------------------------------------- | | Demo and generated apps | Gateway `ext_authz` → oauth2-proxy | The application contains no authentication code | | Argo CD, Backstage, Grafana | Native OIDC client → Keycloak | Each product has its own user and role model | | Gitea | Local `demo` / `demo` account | The PR merge is a separate platform-administration action; this is not SSO | The Gateway admits an application route only from a namespace with the expected environment label. Check one route: ```bash kubectl -n app-demo get httproute demo \ -o jsonpath='{.status.parents[0].conditions[*].type}' ``` Expected result: `Accepted ResolvedRefs`. Without a browser session, the same route redirects to Keycloak: ```bash curl -sk -o /dev/null -w '%{http_code} -> %{redirect_url}\n' \ https://demo.127-0-0-1.sslip.io ``` Expected result: a `302` redirect to `https://auth.127-0-0-1.sslip.io/realms/kensan/...`. ## Why the certificate warning is accurate cert-manager creates two issuers. A bootstrap self-signed issuer creates the `kensan-lab explore` CA; that CA signs one wildcard certificate for the Gateway. TLS encryption is present, but no public authority has verified domain ownership, so browsers correctly refuse to trust it automatically. ``` flowchart LR Self[Self-signed bootstrap issuer] --> CA[Explore root CA] CA --> Wildcard[*.127-0-0-1.sslip.io certificate] Wildcard --> Gateway[Istio Gateway] Browser -. not trusted by default .-> CA ``` The walkthrough gives both honest choices: bypass the warning for a localhost-only demo, or export `explore-ca-tls` and deliberately add that CA to the browser or operating-system trust store. `make try` never changes trust on the host. ## Source change exercise Scaffolding is not the only delivery path the platform owns; the ordinary one is a commit. In the repository the Golden Path created, open `frontend/src/App.tsx`, change one visible sentence with Gitea's edit button and commit to `main`. A second Actions run tests and builds the source, pushes an image tagged with the new commit SHA, and writes that tag into `deploy/values.yaml`. Argo CD sees the tag change and replaces the pod; refresh the page to read the new sentence. The interesting half is the failure. A commit that breaks the tests never reaches the image tag, so `deploy/values.yaml` keeps pointing at the last image that passed and the running pod is never disturbed. Nothing rolls back, because nothing rolled forward. ## GitOps self-healing exercise Scale the original demo by hand: ```bash kubectl -n app-demo scale deploy demo --replicas=3 kubectl -n app-demo get deploy demo -w ``` Argo CD returns it to the Git-declared single replica within a second or two: the controller watches the cluster and does not wait for a Git poll to notice. A direct cluster edit is only a temporary opinion when `selfHeal` is enabled. This is usually over before the next scrape, so the Grafana walkthrough drives CPU with the application's own load button instead of pretending the replica jump will always be graphed. ## Application metrics The **Explore App Runtime** dashboard combines: - container CPU from kubelet/cAdvisor; - desired and available replicas from kube-state-metrics; - `http_requests_total` from the FastAPI service; - p95 latency from `http_request_duration_seconds`. The built-in demo and generated applications each have a `ServiceMonitor` for `/metrics`, scraped every 15 seconds. `metrics-server` is not installed, so `kubectl top` is unavailable; that does not affect Prometheus or Grafana. The CPU panel's resolution is not the Prometheus scrape interval, which is a thing worth knowing before tuning the wrong knob. The kubelet's cAdvisor endpoint is scraped every 10 seconds, but cAdvisor timestamps its own samples and only recomputes per-container statistics about every 15 seconds, so the series advances at that rate however often Prometheus asks. Measured on a running Explore cluster: `up` for that endpoint records 6 samples a minute, `container_cpu_usage_seconds_total` records 4. ## Policy exercise All production `ClusterPolicy` objects run in Audit mode. The generated apps pass because the platform chart supplies security context and resource requests. Create a deliberately non-compliant pod: ```bash kubectl -n app-demo run oops --image=nginx:latest --restart=Never sleep 70 kubectl -n app-demo get policyreport -o json \ | jq -r '.items[].results[] | select(.result=="fail") | "\(.policy): \(.message)"' kubectl -n app-demo delete pod oops ``` You should see the mutable `:latest` tag and missing resource requests reported. Audit records violations without blocking the pod. The one-minute wait is Explore-specific; bare metal scans less frequently to avoid unnecessary etcd writes on constrained hardware. ## Storage substitution Production workloads request `storageClassName: longhorn`. Explore provides the same class name through kind's local-path provisioner so manifests bind without being rewritten: ```bash kubectl get storageclass ``` The output should show both `longhorn (default)` and `standard` using the `rancher.io/local-path` provisioner. This demonstrates the contract, not Longhorn's behavior. There is no replica rebuild, snapshot, backup, expansion, or node failover in a single-node cluster. ## CNI chaining Istio CNI appends itself to kindnet rather than replacing the cluster network: ```bash docker exec kensan-lab-explore-control-plane \ cat /etc/cni/net.d/10-kindnet.conflist \ | jq -r '[.plugins[].type] | join(" ")' ``` Expected result: `ptp portmap istio-cni`. On bare metal, the same setting chains Istio onto Cilium. Explore does not attempt to emulate Cilium L2 announcements on Docker's bridge network. ## What kind cannot demonstrate | Capability | Why it stays on bare metal | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Cilium L2 load-balancer failover | kind nodes are containers on one Docker bridge, not peers on a physical LAN | | Longhorn recovery and node drain | one node cannot lose or rebuild a replica | | Multi-architecture scheduling | every kind node has the host architecture | | Vault root of trust | unseal material and production credentials cannot be transferred to a visitor | | Publicly trusted TLS | the visitor does not own the production domain or its DNS credentials | | Hardware telemetry | temperature, Wi-Fi, disk, and router probes do not exist inside the Docker node | | Production image supply chain | Explore has a local runner and registry, but not GHCR credentials, release retention, signing, or multi-architecture builds | The bare-metal [architecture pages](https://yu-min3.github.io/kensan-lab/architecture/index.md) describe these capabilities. The [bootstrap guide](https://yu-min3.github.io/kensan-lab/bootstrapping/index.md) is a reference for the live cluster, not a clean-room-tested installer; automation is planned. ## Troubleshooting ### A pod is Pending or the node is under memory pressure Give Docker at least 8 GiB, then run `make explore-down && make try`. ### Port 80 or 443 is already in use Use `sudo lsof -nP -iTCP:80 -sTCP:LISTEN` and the equivalent command for 443. A previous Explore cluster is the usual cause. ### An Argo CD Application does not become healthy ```bash kubectl -n argocd describe application ``` Sync waves order Application creation, not every dependency's readiness. Argo CD retries while charts and webhooks converge. ### The browser warns about privacy That is the expected local CA. Follow [Accept the local certificate](https://yu-min3.github.io/kensan-lab/getting-started/try-it-with-kind/#2-accept-the-local-certificate) to bypass it or trust the generated root explicitly. ### Gitea says you cannot merge the pull request That is usually the signed-out view. Sign in with `demo` / `demo`. This local Gitea administrator owns the repository and can merge. ### Everything is healthy but a browser URL does not resolve Some corporate resolvers block wildcard DNS services: ```bash for h in argocd backstage grafana demo auth gitea app2; do \ echo "127.0.0.1 $h.127-0-0-1.sslip.io"; done | sudo tee -a /etc/hosts ``` ### Sign-in ends at `/oauth2/callback` with 403 The CSRF cookie lasts fifteen minutes. Restart in a private window and complete the login without leaving the form open. ### kind cannot pull or parse the node image Upgrade kind. The cluster pins a Kubernetes node image by digest, and v0.32.0 is the tested kind version. # environments/kind — the explore layer A subset of this platform that stands up on a kind cluster in about ten minutes, so that someone can look at how it is composed without owning the hardware. User-facing instructions are in [docs/getting-started/try-it-with-kind.md](https://github.com/yu-min3/kensan-lab/blob/main/docs/getting-started/try-it-with-kind.md); this file is for whoever has to keep it working. ## What is here ```text environments/kind/ kind-cluster.yaml single node, host ports 80/443, hardware-class label versions.sh the one version that cannot live in an Application explore-root-app.yaml app-of-apps entrypoint; takes repo + revision applications/ in-repo Helm chart rendering the explore Applications values/ values layered on top of the bare-metal ones resources/ the substitution layer — the only genuinely new manifests resources-gitea/ the in-cluster git server's namespace and route; applied by the script rather than synced, because Argo CD reads out of the server they describe ``` Argo CD is installed by `scripts/explore-up.sh` with Helm, because nothing can sync it before it exists. Everything after that arrives through `explore-root-app.yaml` and behaves like the real cluster. The script does four other things Argo CD cannot, all for the same reason — they have to happen either before there is a cluster to sync into, or after something Argo CD created is already running: | | why it is not an Application | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **generates every SSO client credential** | a client secret committed to a public repository is a working credential against every cluster started from it; only the disposable `demo` user's password is fixed | | **creates the Keycloak realm with `kcadm`** | Keycloak does not substitute environment variables during realm import — verified: `${VAR}` in a client secret is stored as those six characters — so a realm file in git would have to carry real secrets | | **rewrites CoreDNS** | the issuer hostname has to resolve to the gateway from inside a pod and to `127.0.0.1` from the browser; that is one name with two answers | | **installs Gitea and pushes this checkout into it** | it is the repository Argo CD reads, so it cannot be described by an Application that Argo CD would have to read out of it | Bare metal solves the first two the same way, with `bootstrap/keycloak/setup.sh` run once by hand. Explore runs its own smaller version automatically because there is nobody to run it and nothing worth keeping if it goes wrong. ## Why an Application per component instead of a patch The original design was to reference the bare-metal `app.yaml` files from a kustomization and append a values file with a JSON6902 patch, so no spec would be duplicated. That does not work, and the reasons are worth recording so nobody tries it again: - Kustomize's default load restrictor refuses plain **file** references outside the kustomization root, which is what `../../../kubernetes/argocd/...` is. Directory references are allowed, but only when the directory holds a `kustomization.yaml`. - Putting `kustomization.yaml` files into `kubernetes/argocd/applications/` breaks production: `platform-root` syncs that tree as a **directory** source with `recurse: true`, so it would try to apply `kind: Kustomization` to the API server. - Relaxing the restrictor repo-wide (`--load-restrictor LoadRestrictionsNone` in `argocd-cm`) would weaken a security boundary on the real cluster for the benefit of a demo. So the specs are duplicated, deliberately, and the risk that comes with duplication — a chart version silently drifting apart — is converted into a CI failure instead. ## Anti-drift Every explore Application names the bare-metal Application it mirrors: ```yaml metadata: annotations: kensan-lab.platform/mirrors: kubernetes/argocd/applications/network/istiod/app.yaml ``` `scripts/validate_argocd_apps.py` reads that annotation and fails the build when the upstream chart name, chart repository or version disagree. The annotation may also point at an **ApplicationSet**, which is how `prometheus` and `grafana` are checked: bare metal generates all six observability Applications from one ApplicationSet, whose template says `chart: '{{ .chart }}'` and so cannot be compared directly. The validator follows the generator to the `config.json` the Application is named after and compares against that instead. `versions.sh` is a knowing exception to the repository rule that chart versions live in an Application's `targetRevision`: Argo CD's own chart version is needed before Argo CD exists, so `explore-up.sh` has to read it from somewhere else. The same validator checks that file against the bare-metal Argo CD Application, so the exception cannot drift either. Nothing else belongs in `versions.sh`. What is **not** checked, on purpose: which components explore runs, and what values it layers on. Explore is a subset with substitutions; those are supposed to differ. ## Maintenance 1. **Adding a component to the real cluster does not add it here.** If you want it in explore, add one template under `applications/templates/`. Explore not growing on its own is the feature — it is what keeps this from becoming a second platform to maintain. 1. **Chart versions come from the bare-metal Application.** Never bump one only on the explore side; the drift guard will stop you, and the guard is right. 1. **When Explore CI goes red, ask whether the production change is still reproducible before you touch anything in here.** This job exists to catch changes that quietly acquire a dependency on hardware, on Cilium, or on Vault. Making explore green again without answering that question throws away the signal. 1. **The substitutions are supposed to differ from their bare-metal counterparts.** `gateway-explore.yaml`, `storageclass-longhorn.yaml` and `tls-selfsigned.yaml` are not out-of-date copies. Do not sync them up. `tls-selfsigned.yaml` in particular has no bare-metal counterpart to sync to: production's issuer is a Let's Encrypt `ClusterIssuer` doing a DNS-01 challenge, and the two are alternatives rather than versions of each other. 1. **Nothing explore-related goes in `kubernetes/argocd/applications/`.** `platform-root` syncs that directory with `recurse: true`; a file left there lands on the real cluster. 1. **Do not hardcode a new site-specific value into a raw manifest.** Domains, IPs and issuer URLs are the reason three of the files here exist at all, and nothing is coming to remove them. Every new one is permanent, so this is the single rule that decides whether this layer ages well. 1. **If CI time becomes a problem, shrink the slice rather than moving the job to nightly.** A slow signal on every pull request is worth more than a fast one nobody reads, and "while we are here, let us also run X in explore" is how the slice gets slow. 1. **Promoting Kyverno to Enforce (ADR-012 Phase 3) means revisiting `values/istiod.yaml`.** The trimmed requests there are sized for Audit mode; under Enforce a workload that asks for too little can be rejected outright. 1. **Do not edit `scripts/explore-up.sh` while it is running.** bash reads a script incrementally, so an edit mid-run shifts the offset it is reading from and produces a syntax error on a line that is perfectly valid. It looks like a bug in the script and is not. 1. **A dashboard whose metrics do not exist here does not belong here.** Explore ships one of production's three because the other two read etcd, Cilium and an OTel collector that a single kind node has none of. Half-empty panels do not demonstrate observability; they demonstrate a broken cluster, which is the opposite of the point. The same test applies to scrape jobs — a target that can only ever be down is worse than an absent one. 1. **`explore-up.sh` refuses an existing cluster, and that is not a rough edge to smooth over.** Runtime client credentials are generated fresh on every run; Keycloak reads its own once, at first startup, into an in-memory database. Reusing a cluster therefore leaves configuration and stored state disagreeing. Refusing and pointing at `make explore-down` is the honest option, and it is why the same restart takes the realm away — see the note in the quickstart. 1. **A version tag does not identify a build.** Backstage exists as two GHCR packages, and both once carried a `v0.0.12` with different contents. Explore ran the older one for a while and showed a guest sign-in card behind the SSO gate. Check the package, not just the tag. ## Gitea, and why it is not like the others Everything else here is bare metal with a value swapped. Gitea is not: the production cluster has no such component, and no Application describes it. It exists because the golden path could not be finished without it. Pressing Create in the portal writes to a git server and opens a change against the platform repository — and a visitor holds no credential for the author's GitHub account, so on GitHub that button stops halfway whatever token they bring. With a git server inside the cluster it completes: a repository is created, a pull request is opened, merging it deploys the service. It pays for itself twice. Argo CD reads from it too, so `make try` runs **the checkout in front of you** — no push, no fork, and no fifteen-minute wait to discover you forgot one. Three things about it are easy to get wrong: | | | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **It cannot be an Application** | Argo CD's source is Gitea. A manifest describing Gitea would have to be read out of Gitea before Gitea exists. `scripts/explore-up.sh` installs it, the same way it installs Argo CD | | **It needs the StorageClass early** | It claims a volume before Argo CD has synced anything, so `longhorn` is applied next to the demotion of kind's built-in class rather than left to arrive with the Applications | | **`strategy: Recreate` is required** | The claim is RWO. A rolling update leaves two pods reaching for it and the new one crash-loops until the old one lets go | What it carries is the last commit, not the working tree — `git push HEAD`. Work that is committed but not pushed runs; work that is only saved does not. ## Why the substitutions are permanent Nine of the files here exist because a hostname is hardcoded in a raw manifest — the Gateway, the routes for Argo CD, Backstage, Grafana and Keycloak, the wildcard certificate, Keycloak's env config, the route for Gitea, and the authorization policy naming the hosts it does *not* protect. For a while the plan was to centralise those values behind a `site.yaml` and delete these copies. **That is no longer planned**, so they stay. The reasoning, in short: centralising was justified by making the repository easier to fork, and forking is not what this repository is for. The place somebody tries the platform is here, on kind, where no domain is needed at all (`*.127-0-0-1.sslip.io` resolves for everyone). The bare-metal side stays the author's own configuration. Rewriting the Gateway, the routes, the AuthorizationPolicies and the certificates to be site-driven would have meant a broad refactor of the production network and auth layers — where a missed host means an unauthenticated route and a rejected one means an unreachable service — in exchange for deleting three files from a throwaway cluster. Repositories that do solve this, like [onedr0p/cluster-template](https://github.com/onedr0p/cluster-template), are templates you generate a repository *from*: a single `cluster.toml` and a render step at bootstrap, with plain manifests on the other side. That is a different kind of repository than a reference architecture, and copying its answer here would mean becoming one. So: expect these nine to stay, keep them small, and keep new site-specific values out of raw manifests anyway — rule 6 above still holds, because it is what keeps the count from growing past the hostnames that genuinely need one. ## The one component that is genuinely copied `resources/backstage/deployment.yaml` is the largest single divergence in this directory, and the only Deployment explore owns rather than follows. The bare-metal one reads a PostgreSQL password and a GitHub token from Secrets that External Secrets fills from Vault; without Vault those Secrets never exist and the pod never starts, so the references had to go. Everything not forced to differ is copied verbatim — image, port, probes, resource requests — and the header of that file lists the three deliberate differences. When it drifts from the bare-metal Deployment, that is a bug, not a feature: check the two side by side before changing either. Backstage is also the first workload here that runs an Istio sidecar, because the shared namespace carries `istio-injection: enabled`. Nothing else in the slice joins the mesh data plane. # Architecture # Infrastructure The Git source for every platform component synced by Argo CD. ## Directory layout | Directory | Purpose | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `argocd/` | Argo CD itself (root-app, projects, applications, AppProject, etc.) | | `network/` | CNI (Cilium), Service Mesh (Istio), Gateway API, CoreDNS, Cloudflare Tunnel, NetworkPolicy | | `apps/` | App components under platform management (e.g. `apps/app-kensan` = the new kensan's namespace/PVC/syncthing, etc.) | | `observability/` | Prometheus, Grafana, Loki, Tempo, OTel Collector | | `auth/` | Keycloak (OIDC IdP), oauth2-proxy (Gateway ext_authz), Vault OIDC auth | | `secrets/` | Vault + Vault Config Operator + Vault Database/Transit engines, External Secrets, Sealed Secrets, cert-manager, Reloader | | `storage/` | Longhorn | | `policy/` | Kyverno (policy engine + the ClusterPolicy / PolicyException set) | | `backstage/` | Deploy definitions for Backstage (developer portal) — Pattern B. Source lives at the repo top-level `backstage/` (ADR-018 / ADR-020) | | `kube-system/` | Label / PSA / shared-resource management for the `kube-system` namespace | ## Component layout rules Each component directory's file layout branches into two shapes depending on **whether a `values.yaml` is present**. The deciding question: does the Argo CD app render a Helm chart? ### Pattern A: Helm-chart-based (`values.yaml` + `resources/`) Components where an Argo CD multi-source app combines an **upstream Helm chart + values.yaml + additional raw YAML**. ```text / ├── values.yaml # Helm values └── resources/ # Raw YAML outside the chart (namespace, ServiceMonitor, HTTPRoute, SealedSecret, etc.) └── *.yaml ``` Examples: `secrets/vault/`, `network/cilium/`, `observability/grafana/`, `observability/prometheus/`, `secrets/cert-manager/`, `secrets/external-secrets/`, `secrets/vault-config-operator/`, `observability/otel-collector/`, `observability/loki/`, `observability/tempo/` `resources/` exists to **separate out the YAML that Helm doesn't render**. It only makes sense alongside a `values.yaml`, so don't create it for components that have none. ### Pattern B: raw YAML only (flat) Components with no Helm chart, where the Argo CD app reads raw manifests directly. ```text / └── *.yaml ``` Examples: `network/network-policy/`, `network/cloudflare-tunnel/`, `network/gateway-api/`, `secrets/sealed-secrets/`, and the Gateway / PeerAuthentication / namespace, etc. directly under `network/istio/` Place `.yaml` files directly at the top level. No `resources/` indirection. ## Exceptional structures - **`istio/`**: a multi-component directory bundling several charts (`base/`, `cni/`, `istiod/`). Each subchart follows Pattern A with its own `/values.yaml`. Raw YAML that belongs to Istio as a whole but sits outside any chart (Gateway, PeerAuthentication, the `istio-system` namespace, etc.) is placed flat directly under `istio/` (Pattern B). - **`network-policy/`**: a special component that aggregates NetworkPolicy / CiliumClusterwideNetworkPolicy across namespaces. To keep PE-exclusive resources in one place, these are split out from each namespace-owning app and consolidated into the `network-policy` app. - **`observability/`**: has no `app.yaml`. The ApplicationSet at `argocd/applications/observability/applicationset.yaml` uses a git generator to pick up each `observability/*/config.json` (which holds the chartVersion etc.) and generates one Application per component. Each `config.json` is the source of truth for that component's chart version. - **`secrets/vault-database-engine/`, `secrets/vault-transit-engine/`**: a three-part structure of a homegrown chart (`chart/`) + raw YAML shared across all instances (`shared/`) + per-instance values (`platform-values/`). Configured as a pair — `argocd/applications/secrets//app-shared.yaml` (syncs `shared/`) and `applicationset-instances.yaml` (an ApplicationSet that generates one instance per file under `platform-values/`) — with no single `app.yaml`. ## Namespace lifecycle management Most components carry `resources/namespace.yaml` (Pattern A) or a flat `namespace.yaml` (Pattern B) inside their own Application and self-contain namespace creation + labeling via `CreateNamespace=true` (e.g. `secrets/cert-manager/`, `secrets/external-secrets/`, `secrets/reloader/`, `backstage/`, `network/cloudflare-tunnel/`). A dedicated ns-lifecycle app under `kubernetes/argocd/applications/namespaces/` only exists when one of these three patterns applies (ADR-020): | Pattern | Reason | Example | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | **No owning component** | No component in this repo corresponds to the namespace (e.g. one created by kubeadm outside GitOps, adopted here purely to attach labels) | `kube-system` | | **Multiple components share the namespace** | No single owning Application can be designated (`policy/kyverno` + `policy/kyverno-policies` share `policy/namespace.yaml`; the 6 `observability/*` components share `observability/namespace.yaml`) | `kyverno`, `monitoring` | | **Deliberate lifecycle isolation** | There is a single owner, but the app is deliberately split off for protection (e.g. `sealed-secrets` — no finalizer + `Prune=false` to protect the sealing key) | `sealed-secrets` | Cases that are "single-owner but split off" without matching one of the above are consolidation candidates (`reloader` was consolidated in 2026-07, per ADR-020). ## Argo CD Application placement Each component's Argo CD Application CR lives at `argocd/applications///app.yaml`. `platform-root` (the App-of-Apps) recurses `argocd/applications/` to discover and manage every child app. ## Related documents - `.claude/rules/helm-multisource.md`: details of Pattern A (the 3-file structure for Argo CD multi-source) - `.claude/rules/gitops-workflow.md`: GitOps operating rules - `docs/`: ADRs, architecture diagrams # Argo CD: one root app, Git as the only actor The GitOps control plane — Argo CD itself, plus the declarations of everything it manages — built around one idea: **a single root Application discovers the whole platform from Git, so "deployed" and "in Git" can never drift apart.** **Design thesis:** **App-of-Apps with one root, mixed patterns below it.** `platform-root` recursively scans `applications/` and creates every child Application — adding a component to the platform is adding a directory, not touching Argo CD. Below the root, each category deliberately picks between individual Applications and ApplicationSets ([ADR-003](https://github.com/yu-min3/kensan-lab/blob/main/docs/adr/003-applicationset-migration-strategy.md)): ApplicationSet only where structure is uniform enough to stamp out, individual Applications wherever sync-waves or `ignoreDifferences` are bespoke. **What you'll find here:** how the App-of-Apps tree hangs together, where Application CRs live relative to their manifests, and the criteria for choosing ApplicationSet vs individual Applications — a decision you'll face in any GitOps repo that grows past ten components. ## Components | dir / file | role | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `values.yaml` | Argo CD chart overrides — Keycloak OIDC login, server-side diff, multi-source support, `ignoreDifferences` defaults | | `resources/` | Chart-external manifests: argocd ns, `app-project` AppProject, HTTPRoute, OIDC ExternalSecret | | `projects/platform-project.yaml` | AppProject for infrastructure (PE-owned namespaces only) | | `root-apps/platform-root-app.yaml` | The single App-of-Apps entry point — recurses `applications/`, prune + selfHeal on | | `applications/` | Application / ApplicationSet CRs per category — see [`applications/README.md`](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/argocd/applications/README.md) | ## App-of-Apps structure ``` %%{init: {"flowchart": {"htmlLabels": true}} }%% flowchart TB ROOT["platform-root
recurses applications/ | prune + selfHeal"] ROOT --> NET["network/
cilium, istio, gateway-api …
individual apps (sync-waves)"] ROOT --> SEC["secrets/
vault, ESO, cert-manager …
individual + 2 engine ApplicationSets"] ROOT --> AUTH["auth/
keycloak, oauth2-proxy …
individual apps"] ROOT --> OBS["observability/
one ApplicationSet,
stamped from config.json
"] ROOT --> MORE["namespaces/ | storage/ | policy/
backstage/ | gitops/
individual apps"] ROOT --> APPS["apps/
user apps —
auto-committed by Backstage"] classDef default fill:#26221D,stroke:#4A4232,color:#FCFAF6 classDef root fill:#8A4B12,stroke:#FDBA74,color:#FCFAF6 classDef aset fill:#075985,stroke:#38BDF8,color:#FCFAF6 classDef user fill:#2F6B45,stroke:#7AC99A,color:#FCFAF6 class ROOT root class OBS aset class APPS user ``` Application CRs live in `applications///app.yaml`; the manifests they deploy live in `kubernetes///`. Layout rules (Pattern A/B, multi-source): [`kubernetes/README.md`](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/README.md). ## Design rationale **Three principles thread the whole design:** 1. **One root, everything discovered.** `platform-root` is the only manually-applied Application; everything else is found by recursion. With `prune: true` + `selfHeal: true`, deleting a directory deletes the deployment and manual `kubectl` edits revert themselves — Git is not *a* source of truth, it is the only actor. 1. **ApplicationSet only where structure is uniform.** Stamping requires uniformity; forcing it everywhere trades away per-app control (sync-waves, `ignoreDifferences`) for cosmetic DRY. So only three uses exist: observability (identical Helm multi-source components, parameterized by `config.json`) and the two Vault engines (per-instance `platform-values/`). Everything else stays as readable individual Applications — the full criteria and per-category table are in [ADR-003](https://github.com/yu-min3/kensan-lab/blob/main/docs/adr/003-applicationset-migration-strategy.md) and [`applications/README.md`](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/argocd/applications/README.md). 1. **Render in Argo CD, never in Git.** Applications reference upstream charts + `values.yaml` + `resources/` as multiple sources; Argo CD renders natively. No `helm template` output is ever committed, so diffs stay reviewable and upgrades are a one-line `targetRevision` bump. Concrete choices: - **AppProjects are the blast-radius boundary.** `platform-project` (PE-owned infrastructure namespaces) and `app-project` (AD-owned app namespaces) cap what any Application may touch — a compromised or misconfigured app CR cannot deploy into `kube-system`. - **The AD path is fully automated.** Backstage scaffolds a new app repo and auto-commits its Application CR into `applications/apps/` — developers ship to the cluster without ever editing this directory by hand. - **Argo CD manages itself** (`gitops/` category): its own chart version, OIDC config, and HTTPRoute follow the same PR flow as everything it deploys. - **Server-side diff + targeted `ignoreDifferences`** keep noisy fields (mutating webhooks, operator-written status) from producing phantom OutOfSync — hard-won details like "never write a lone `directory.recurse: false`" are recorded in [`applications/README.md`](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/argocd/applications/README.md). ## Related - [ADR-003 ApplicationSet migration strategy](https://github.com/yu-min3/kensan-lab/blob/main/docs/adr/003-applicationset-migration-strategy.md) — per-category pattern choice + addendum - Component layout rules (Pattern A/B, multi-source 3-file structure): [`kubernetes/README.md`](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/README.md) - GitOps workflow (worktrees, verification exception, prune protection): [`.claude/rules/gitops-workflow.md`](https://github.com/yu-min3/kensan-lab/blob/main/.claude/rules/gitops-workflow.md) # Golden Path: shipping an app via Backstage The end-to-end walkthrough of the platform's fully-automated App Developer (AD) path: **one Backstage form → a running, observable, SSO-protected application** — without the developer ever editing platform YAML. This is the human-facing counterpart to the architecture page ([`kubernetes/backstage/README.md`](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/backstage/README.md)) and the role model ([PE / AD split](https://github.com/yu-min3/kensan-lab/blob/main/.claude/rules/environment-separation.md)). ``` %%{init: {"flowchart": {"htmlLabels": true}} }%% flowchart LR AD["App Developer"] -->|one form| BS["Backstage template"] BS --> REPO["App repository
code · CI · manifests · TechDocs"] BS --> PR["Platform PR
Argo CD Application"] BS --> CAT["Catalog entry"] PR -->|PE review + merge| ARGO["Argo CD"] REPO --> ARGO ARGO --> LIVE["Running app
SSO · zero trust · observability"] ``` ## What one form submission produces | Artifact | Where | Owner | | ------------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------ | | App repository (code, Dockerfile, CI, `manifests/`, TechDocs) | `github.com//` (new) | AD | | Argo CD `Application` CR | PR against `kensan-lab` → `kubernetes/argocd/applications/apps//` | PE reviews, then Argo CD | | Catalog entry + TechDocs | Backstage (`catalog-info.yaml` auto-registered) | AD | A live catalog entry on this platform: the component is discoverable with ownership, system relations, source, and TechDocs in one place. ## Walkthrough ### 1. Create from the template Open Backstage (`backstage.platform.yu-min3.com` — SSO happens at the Gateway, no separate Backstage login) → **Create…** → **FastAPI Application**. Fill in: - **Application Name** — lowercase / digits / hyphens. Becomes the Argo CD Application name (`app-`), the catalog component name, and the image name. - **Owner** — picked from the catalog (Group/User). - **Repository Location** — where the new app repo is created. Argo CD authenticates to any repo under the org prefix via credential templates, so no per-repo secret setup is needed. - **Domain / System** — catalog placement ([`backstage/catalog/`](https://github.com/yu-min3/kensan-lab/tree/main/backstage/catalog) defines the domains). ### 2. The scaffolder runs three actions 1. **`publish:github`** — creates the app repository from the skeleton: FastAPI service, Dockerfile, a GitHub Actions build-and-push workflow (GHCR), flat `manifests/` (Deployment / Service / HTTPRoute / AuthorizationPolicy / ServiceMonitor), `catalog-info.yaml`, and an MkDocs TechDocs skeleton. 1. **`publish:github:pull-request`** — opens a PR against `kensan-lab` adding the Application CR under `kubernetes/argocd/applications/apps//`. Deliberately a PR, not a direct commit: the platform keeps its review gate even on the automated path. 1. **`catalog:register`** — the new component appears in the service catalog with TechDocs. ### 3. Platform Engineer merges the PR Review checklist is short by construction — the CR is generated, scoped to `app-project`, and can only deploy into AD-owned namespaces. On merge, `platform-root` (App-of-Apps) discovers the new Application automatically; no Argo CD configuration is touched. ### 4. The app goes live Argo CD syncs the app repo's `manifests/` into the app namespace. The generated manifests already integrate the platform: - **Ingress + SSO** — HTTPRoute attaches to the shared Gateway; OIDC is enforced at the edge (oauth2-proxy ext_authz), so the app trusts authenticated headers and implements no login. - **Zero-trust defaults** — the namespace baseline (default-deny NetworkPolicy, PSA, Istio mTLS) applies without app-side configuration; the skeleton ships the explicit `AuthorizationPolicy` it needs. - **Observability** — a ServiceMonitor is included; telemetry goes to the central OTel Collector ([integration guide](https://yu-min3.github.io/kensan-lab/runbooks/observability-integration/index.md)). ### 5. Iterate Day-2 development never re-enters the platform repo: push code → CI builds a multi-arch image to GHCR → bump the image tag in the app repo's `manifests/` → Argo CD syncs. Secrets, when needed, are requested from the platform rails ([secret decision framework](https://yu-min3.github.io/kensan-lab/secret-management/index.md)). ## Current state and target model The namespace model is one namespace per app (`app-`, [ADR-006](https://yu-min3.github.io/kensan-lab/adr/006-namespace-naming/index.md) / [ADR-014](https://yu-min3.github.io/kensan-lab/adr/014-namespace-naming-label-contract-v2/index.md)) with the namespace and HTTPRoute owned by the app side — `app-kensan` runs this pattern, and the shared `app-prod` landing zone was retired with the migration (2026-07). The scaffolder template still generates a CR targeting the retired `app-prod` namespace and legacy dev/prod wording; updating it to the per-app model is a known follow-up, so treat the template's namespace output as pending that update. # cloudflare-tunnel Outbound-only public ingress. `cloudflared` dials **out** to Cloudflare's edge — no port-forward, no NodePort, no public IP on the cluster. Cloudflare Access sits in front as an edge auth gate, and traffic is delivered to the in-cluster **Istio Gateway** — the same chokepoint LAN traffic uses (see the [network overview](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/network/README.md)). ## In-cluster components | file | role | | ---------------------- | ------------------------------------------------------------------------------------------------------------------ | | `deployment.yaml` | `cloudflared` Deployment (token-managed tunnel, image `cloudflare/cloudflared:2026.10.4`, metrics on `:2000`) | | `external-secret.yaml` | ESO pulls the tunnel token from Vault (`secret/cloudflare-tunnel/token`) into the `cloudflare-tunnel-token` Secret | | `namespace.yaml` | the `cloudflare-tunnel` namespace | The tunnel is **token / dashboard-managed**: the pod runs `cloudflared tunnel run` with a `TUNNEL_TOKEN`. Routing (public hostnames) and Access policies live in the **Cloudflare Zero Trust dashboard**, not in Git — the manifests only run the connector. > Step-by-step walkthrough (JP): [自宅からポート開放なしで認証付きアプリを公開 — Cloudflare Tunnel + Access](https://zenn.dev/yuu7751/articles/9df7ce4f1f4830). This README documents how the pattern is wired in kensan-lab specifically (token via Vault/ESO, routed to the Istio Gateway). ## How traffic gets in ``` %%{init: {"flowchart": {"htmlLabels": true}} }%% flowchart LR Net([Internet]) -->|HTTPS| CF["Cloudflare edge
TLS termination + DDoS"] CF --> ACC{"Cloudflare Access
OTP gate"} ACC -->|allowed| CFD["cloudflared
(cloudflare-tunnel ns)"] CFD --> GW["Istio Gateway
(istio-system svc)"] GW --> APP["app"] classDef default fill:#26221D,stroke:#4A4232,color:#FCFAF6 classDef ext fill:#8A4B12,stroke:#FDBA74,color:#FCFAF6 classDef gate fill:#075985,stroke:#38BDF8,color:#FCFAF6 class Net,CF ext class ACC,GW gate ``` The connection between `cloudflared` and Cloudflare is **dialed outbound by the pod** (QUIC), so nothing needs to be opened inbound at the router. Traffic then rides that established tunnel back down to the Gateway. ## Tunnel configuration (dashboard) Networking → **Tunnels** → create a tunnel → copy the token (stored in Vault, below). Then add a **Public Hostname**: | field | value | | ------------------ | ------------------------------------------------------------------ | | Subdomain / Domain | e.g. `kensan` / `yu-mins.com` | | Type | `HTTP` | | URL | `http://.istio-system.svc.cluster.local:80` | Point the URL at the **in-cluster Istio Gateway service** (not directly at an app), so external and LAN ingress converge on one Gateway that owns TLS re-termination, routing, and Keycloak auth. **Token handling (GitOps-friendly):** the token lives in Vault at `secret/cloudflare-tunnel/token`; ESO syncs it into the `cloudflare-tunnel-token` Secret (`refreshInterval: 1h`), which the Deployment consumes as `TUNNEL_TOKEN`. Rotating the tunnel = update Vault → ESO resyncs. **NetworkPolicy:** the `cloudflare-tunnel` ns is not istio-injected, so it uses a per-ns NetworkPolicy (`kubernetes/network/network-policy/cloudflare-tunnel.yaml`, see the [network-policy guide](https://github.com/yu-min3/kensan-lab/blob/main/docs/architecture/network-policy-guide.md)). `cloudflared` needs egress for: DNS (53), Cloudflare edge over **QUIC (UDP 7844)** with TCP 443 fallback, and the Istio Gateway service. ## Access configuration (dashboard) Zero Trust → **Access** → Applications → **Self-hosted**: | field | value | | ------------------ | -------------------------------------------------------------------------------------- | | Application domain | the tunnel's public hostname (e.g. `kensan.yu-mins.com`) — must match the Tunnel route | | Session Duration | e.g. `24h` (independent of the inner Keycloak session) | **Policy:** `Action: Allow`, selector `Emails` = your address, delivered via **Email OTP**. Access is an **edge gate**, not the only auth: unauthenticated requests die at Cloudflare's edge before entering the tunnel, while the in-cluster Istio Gateway still enforces Keycloak auth on every route. The current interim model is OTP in front of `*.yu-mins.com`; whether to keep Access as a second factor, collapse it into Keycloak OIDC, or drop it for pass-through is the deferred decision in [ADR-016](https://github.com/yu-min3/kensan-lab/blob/main/docs/adr/016-lan-frictionless-cf-access-external-gate.md). ## Reference values | element | value | | ------------------ | ----------------------------------------------- | | external domain | `*.yu-mins.com` (Cloudflare Tunnel) | | namespace / Secret | `cloudflare-tunnel` / `cloudflare-tunnel-token` | | token Vault path | `secret/cloudflare-tunnel/token` | | connector image | `cloudflare/cloudflared:2026.10.4` | ## Related - Network overview and the two ingress paths: [network README](https://github.com/yu-min3/kensan-lab/blob/main/kubernetes/network/README.md) - External-gate model (Access vs Keycloak-only vs pass-through): [ADR-016](https://github.com/yu-min3/kensan-lab/blob/main/docs/adr/016-lan-frictionless-cf-access-external-gate.md) - Cloudflare-side edge, LB IPs, known issues: [`.claude/rules/network-ingress.md`](https://github.com/yu-min3/kensan-lab/blob/main/.claude/rules/network-ingress.md) - Cloudflare security advisory: [eMudhra certificate compromise](https://blog.cloudflare.com/emudhra-certificate-compromise/) # Cluster Health Monitoring design A design for surfacing kensan-lab's health (node liveness, resources, endpoint reachability) in a single Grafana view and notifying Slack on anomalies. Builds in three phases on top of the existing observability stack ([observability.md](https://yu-min3.github.io/kensan-lab/architecture/observability/index.md)). ## Background and gaps The existing stack **already has the data path and alert path in place**: - node-exporter (DaemonSet) + kube-state-metrics → Prometheus (7d retention) - kube-prometheus-stack's defaultRules (`node` / `nodeExporterAlerting` enabled) → NodeNotReady and similar already notify Slack's `#k8s-alerts` - Grafana's sidecar auto-imports ConfigMap dashboards (existing: controlplane / longhorn / claude-code / otel-apm) What's missing: | # | Gap | Phase | | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | 1 | No single dashboard for overall cluster health (node liveness, temperature, remaining microSD capacity, wired/WiFi state) | Phase 1 | | 2 | Liveness monitoring only means "could Prometheus scrape it." Can't distinguish a dead kubelet from a dead OS, or a wired-link drop from a total outage. HTTP reachability of each UI through the Gateway is also unmonitored | Phase 2 | | 3 | **The monitoring stack itself is an m4neo SPOF**. Prometheus / Alertmanager / Grafana all require `hardware-class=high-performance`, which pins them to m4neo. If m4neo dies, monitoring goes silent along with it — no alert fires | Phase 3 | ## Phase 1 — Cluster Health dashboard No deployment changes. Adds a single dashboard using only existing metrics. - **Location**: `kubernetes/observability/grafana/resources/cluster-health-dashboard.yaml` (a ConfigMap with `grafana_dashboard: "1"` + `grafana_folder: "Platform"`; same shape as the controlplane-dashboard) - **Approach**: hand-built rather than importing a community dashboard (e.g. Node Exporter Full) — tuned to a density where a 4-node homelab is legible at a glance ### Panel layout **Row 1: Node liveness** | Panel | Query | Type | | ------------------------ | -------------------------------------------------------------------------------------------------- | -------------------------- | | Node Ready | `kube_node_status_condition{condition="Ready",status="true"}` | stat (per node, green/red) | | node-exporter up history | `up{job="node-exporter"}` | state timeline | | Node Uptime | `node_time_seconds - node_boot_time_seconds` | stat | | Pressure conditions | `kube_node_status_condition{condition=~"MemoryPressure\|DiskPressure\|PIDPressure",status="true"}` | table / stat | **Row 2: Resources** | Panel | Query | Notes | | ----------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | CPU usage | `1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))` | Per-node timeseries | | Memory usage | `1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes` | Per-node timeseries | | Disk usage (root) | `1 - node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} / node_filesystem_size_bytes{...}` | **Primary purpose: catching microSD exhaustion on the Pis**. Thresholds: yellow at 80%, red at 90% | | CPU temperature | `node_hwmon_temp_celsius` (falls back to `node_thermal_zone_temp` if unavailable on the Pis) | Watches for thermal throttling on the Pis (80°C threshold) | **Row 3: Network (visualizing the wired/WiFi dual-path setup)** | Panel | Query | Notes | | ----------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------- | | Wired link state | `node_network_carrier{device=~"eth0\|eno1\|enp4s0"}` | Detects a wired drop that fell back to WiFi | | WiFi link state | `node_network_carrier{device=~"wlan0\|wlp3s0"}` | | | Interface traffic | `rate(node_network_receive_bytes_total{device=~"eth.*\|en.*\|wlan.*\|wlp.*"}[5m])` | Shows which path traffic is actually flowing over | **Row 4: Cluster overview** | Panel | Query | Notes | | ------------------ | --------------------------------------------------------------------------- | ----------------------------------- | | Abnormal pod count | `sum(kube_pod_status_phase{phase=~"Pending\|Failed\|Unknown"})` | stat, green at 0 | | Container restarts | `sum by(namespace)(increase(kube_pod_container_status_restarts_total[1h]))` | Early detection of CrashLoops | | PVC usage | `kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes` | Covers both Longhorn and local-path | | Pods per node | `sum by(node)(kube_pod_info)` | Checking for scheduling skew | Things to confirm during implementation: - Confirm the actual temperature metric name on the Pis (`node_hwmon_temp_celsius` vs `node_thermal_zone_temp`) against live Prometheus - Confirm `node_network_carrier` is available on the WiFi interfaces (fall back to `node_network_up` if not) ## Phase 2 — blackbox-exporter (probe-based liveness monitoring) Adds active ICMP / HTTP probes on top of Prometheus's internal-view scraping. - **Location**: `kubernetes/observability/blackbox-exporter/` (`config.json` + `values.yaml`). Auto-discovered by the observability ApplicationSet (git file generator) - **Chart**: `prometheus-community/prometheus-blackbox-exporter` - **Note**: ICMP probes need the `NET_RAW` capability (granted via the chart's `securityContext`) ### Probe targets Defined via the Prometheus Operator's `Probe` CRD (placed in `resources/`): | Type | Target | What it reveals | | ---- | -------------------------------------------------------------------------- | ------------------------------------------------------------ | | ICMP | Wired IPs: 192.168.0.107–110 | OS-level liveness (distinguishes this from a dead kubelet) | | ICMP | WiFi IPs: 192.168.0.207–210 | Whether the WiFi fallback is alive when the wired link drops | | HTTP | `https://{grafana,argocd,backstage,vault,prometheus}.platform.yu-min3.com` | End-to-end reachability through Gateway → cert → backend | ### New alerts (PrometheusRule) | Alert | Condition | Severity | | -------------------- | ------------------------------------------------------------- | ----------------------------- | | NodeICMPUnreachable | Both wired and WiFi ICMP probes fail, sustained 5 minutes | critical | | NodeWiredLinkDown | Wired probe fails & WiFi probe succeeds, sustained 10 minutes | warning (running on fallback) | | PlatformEndpointDown | HTTP probe fails, sustained 5 minutes | warning | Add a probe-results row to the dashboard (update the Phase 1 JSON). ## Phase 3 — Grafana Cloud remote_write (mitigating the m4neo SPOF) Puts "monitoring the monitoring, plus a dashboard that stays visible during an outage" outside the cluster. Changed from the original plan of healthchecks.io (Watchdog forwarding only) to a **filtered remote_write into Grafana Cloud's free tier** — this adds a dead-man's-switch (no-data alert) plus the ability to "view metrics from just before a total cluster outage, from outside the cluster." - **Mechanism**: local Prometheus scraping stays entirely in-cluster as before; `remote_write` (outbound HTTPS push, no inbound needed) replicates a carefully-chosen subset of series to Mimir on Grafana Cloud (Tokyo region) - **Free tier**: 10k active series / 14-day retention / 3 users. The allowlist measures out to ~1.4k series (14%) in practice - **Implementation**: - Create a Grafana Cloud account + stack (manual, no credit card required). Issue a remote_write token (`metrics:write` scope) into Vault static (`secret/monitoring/grafana-cloud/remote-write`) - Sync it into a `grafana-cloud-remote-write` Secret (`monitoring` namespace) via ESO - Add the endpoint + basicAuth + `writeRelabelConfigs` to `prometheusSpec.remoteWrite` (keep the cluster-health dashboard's series, drop non-idle CPU modes) - Replicate the Cluster Health dashboard on the Cloud side (only the datasource uid needs to change) - Set up a **no-data alert** on the Cloud side (`up{job="node-exporter"}` no longer arriving) — this is the dead-man's-switch - **Failure modes covered**: m4neo going down, Prometheus / Alertmanager stopping, a total cluster outage, home network / ISP outage - **Candidate Phase 4 (optional)**: an external HTTP probe via Cloudflare Workers cron + a public status page (to distinguish a home-ISP outage from a Gateway failure from an outside vantage point) ## PR split | PR | Content | Deploy impact | | --- | ----------------------------------------------------------------- | ------------------------------ | | 1 | This design doc + Phase 1 dashboard | None (ConfigMap addition only) | | 2 | Phase 2 blackbox-exporter + Probes + alerts + dashboard update | One new Pod (Deployment) | | 3 | Phase 3 Grafana Cloud remote_write (ExternalSecret + values.yaml) | Prometheus config reload | ## Related - Overview of the existing stack: [observability.md](https://yu-min3.github.io/kensan-lab/architecture/observability/index.md) - Node topology and labels: `.claude/rules/kubernetes-cluster.md` - The wired/WiFi dual-path setup: [network.md](https://yu-min3.github.io/kensan-lab/architecture/network/index.md) ## 結論 工房はprivate `yu-min3/koubou` repoのapp-base用valuesと追加resourcesで配備する。kensan-labにはapp-projectのApplication登録だけを置き、共通Deployment/Service/HTTPRouteを再実装しない。共有DNS・Gateway・SSOの変更をこのPRに混ぜない。配備は手動syncとし、以下の前提が満たされるまで運転開始しない。 旧PR案のnetwork/koubou-entry、platform-project、共有Gateway認可へのhost追加、工房専用DestinationRule/SealedSecret、新LAN DNSは今回のdiffから削除した。旧案は未マージ・未配備であり、既存リソースのpruneは発生しない。 ## App/Platformの契約 | 項目 | Appが所有 | Platformが提供 | | -------- | ------------------------------------------------------------------ | ------------------------------------------------- | | 配備 | private repoのchart/valuesと固定commit | Argo CD、app-project、repo参照権限 | | 通信 | app-koubou内relay/Service/route/NetworkPolicy、senseへのCA/SAN検証 | gateway-prodのHTTPS・既定SSO、許可されたroute接続 | | 認証 | senseの本人sub/group/署名検証、内部session | Keycloak本人認証、署名済みID token、callback登録 | | 名前解決 | 必要なhostとGateway IPを要求 | LAN端末のDNS解決。DHCPや共通DNSはApp管理外 | namespaceはApp resourcesが所有しPrune=false。PVCなし。App内relayからsenseへTLSを張るためEndpointSliceやGateway用DestinationRuleを追加しない。既存SSO契約を使うため共有AuthorizationPolicyに工房hostを追記しない。 候補URL: `https://koubou.app.yu-min3.com`(LAN Gateway 192.168.0.243)。まだ利用不可。App側の仕様・導入手順はprivate repoの `deploy/app-entry.md`。 ## Platformに足りない要素・確認事項 | ID | 現状/不足 | 当面の導入条件 | 継続改善の契約 | 担当 | | ---- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | --------------------------------- | | P001 | private repoの参照権限は既存org credential templateの仕組みがあるが、工房repoへの実アクセス未検証 | Argo CDが工房の固定commitを読めることを確認。資格値は取得/表示しない | repo登録時に接続確認まで返す | Platform | | P002 | Keycloakのcallback登録はhostごとのPlatform操作 | 工房App hostのcallback/weborigin登録を別Platform変更として行う | Appがhostを申請し、Platformが承認/登録/検証結果を返す。共有realmをAppが変更しない | Platform、公開/認証変更の判断はYu | | P003 | OAuth2 routeのReferenceGrantがapp-kensan/app-konroを個別列挙 | app-koubouのHTTPRoute→auth-system/oauth2-proxy参照を別変更で許可 | App登録とcross-namespace認証参照許可を一緒に処理する。全namespaceへの無条件許可にはしない | Platform | | P004 | LANスマホ用の名前解決がなく、Mac hostsに依存 | 工房host→.243と認証hostのLAN解決を提供 | LAN DNSを共有基盤として設計・独立PRで提供。Appはレコード要求だけを渡す | Platform、DHCP切替はYu | | P005 | app-baseにcommand/argsと追加volume/mountの指定がない | 共通chart PR #543はレビュー/CI後にマージ済み | Appが共通templateを複製せずvaluesで設定できる | Platform | | P006 | 既存App Gatewayが検証済みJWTを除去し、senseへ本人の署名が届かない | draft PR #544でJWT転送契約とP002/P003の登録を分離。Yu判断後に導入 | Appへ署名済み本人IDを渡し、App側でも検証できる | Platform、共有SSOの判断はYu | P001は機能がないと断定せず確認事項。P002/P003は既存の手動登録契約が残っている。P004がスマホ導入の不足。これらを工房Appが直接作る構成にはしない。外部host TLSはAppのConfigMapで設定する。app-baseに必要な汎用機能はP005として別PRに分離する。 ## 採用・却下 | 判定 | 案 | 理由 | | ---- | --------------------------------------------------------- | ------------------------------------------------------------------------------ | | 採用 | private app-base用valuesと追加resources+既存gateway-prod | Appの追加で共有認可のhost列挙を編集しない。App資源はapp-project内に収まる | | 採用 | App relayがsense TLSを検証 | App固有CAをAppへ閉じ、EndpointSlice権限の拡大や共有DestinationRuleを不要にする | | 却下 | 工房専用入口をnetwork/platform-projectへ置く | AppがPlatform権限に依存し、repo独立の要件を満たさない | | 却下 | このApp PRでLAN DNS/DHCPまで導入する | 宅内全体への影響をAppの変更と分離して判断できない | ## 検証と残作業 既存app-baseと工房valuesはHelm lint/render、schema検証、上流imageのamd64/arm64 manifest確認を実施。Platform登録はAppProjectの許可kind/namespaceと整合する。実SSO・sense・Gateway datapath・スマホは未検証。P001〜P006とsense側導入を満たしてから手動syncし、本人ログイン/登録/停止と拒否経路を実測する。 ## 壊れうるもの/戻し方 このPRの旧infra案は未配備のため、その削除による既存namespace/PVC/DNSのpruneはない。新Appはauto-sync/finalizerなし、ApplicationとnamespaceにはPrune=false。手動配備後にAppを止める場合はApp内route/relayだけを対象とし、共有Gateway/SSO/DNSは削除しない。 ## Yuの未決事項 このdraft PRでは共有Platform変更の承認を求めない。不足要素を別のPlatform改善として具体化してからレビューする。マージ前の本人確認条件を維持する。 ## 共通Helm chartの利用 工房側の独自Deployment/Service/HTTPRoute templatesは削除した。Argo CDは既存app-baseとprivate工房values/resourcesをmulti-sourceで組み合わせる。汎用chartの不足(起動指定、追加mount、API token制御、listener指定)は別draft PR #543で扱う。既存kensan/konro/canaryのrenderが旧chartと同じ構造であることを確認した。工房固有のConfigMap・Namespace・NetworkPolicyだけがprivate repoに残る。 ## 導入前レビューとsense実測 独立レビューはchart/multi-source/selectors/TLS設定に確定P0/P1なし。P006は [Istio 1.27.3実装](https://raw.githubusercontent.com/istio/istio/1.27.3/pilot/pkg/security/authn/policy_applier.go) と [Envoy JWT契約](https://raw.githubusercontent.com/envoyproxy/envoy/v1.35.2/api/envoy/extensions/filters/http/jwt_authn/v3/config.proto) で確認した。PR544の共有認証変更はこのApp登録PRと分離し、未適用。 senseのTLS adapterはlan-sso-v3でactive/running、公開CA/SAN検証と未認証401を実測。systemdのroot:root0440 credential形式を限定許可する修正・拒否試験を追加し、installer成功条件も実TLS応答へ変更。これはSSO全経路の合格ではない。private repoの直接取得とLAN DNSは未確認/未提供の条件として残る。 ## 2026-10-10 導入条件の更新 | 条件 | 現在の状態 | 残る確認 | | ---------------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- | | 共通chart・JWT転送・認証参照 | PR543/544はマージ済み。chartは62cabffeへ固定 | GitOps実反映と本人SSO | | 宅内DNS | .247が稼働。YuがDHCPにPrimary .247/Secondary .1を設定、Mac反映と内部/外部UDP/TCP解決を実測 | 工房route作成後の名前解決とスマホ確認。障害時切替は未検証 | | sense本人検証 | koubou-lan-sso.service稼働、TLS8790。sense限定hostsでauth.yu-mins.comを.242へ解決し、TLS検証付きJWKS取得成功 | 実際の本人JWTで拒否/許可を確認 | | private repo取得 | 専用read-only deploy keyとSealedSecretをPR545へ準備。App参照はSSHへ変更 | Yuの承認後に公開鍵登録、PR545反映、固定commit取得 | 公開DNSの変更は不要。App routeを共通chartから作るとLAN DNSに自動登録される。Appの手動syncはPR545と本PRの承認・GitOps反映確認後に実施する。家族の端末で障害時のDNS切替がどう動くかは別途確認が必要。 # Namespace Label Design This document defines a unified namespace labeling strategy across the entire platform. ## Design Principles 1. **Consistency**: Use the same label key and value patterns across all namespaces 1. **Discoverability**: Easily filterable with `kubectl get ns -l