Skip to content

How the kind environment works

The Explore environment is a tested slice of the platform with explicit substitutions. It is built for learning and pull-request validation; it does not pretend that one laptop reproduces bare-metal networking, storage, or failure behavior.

Follow the walkthrough first. This page explains what it did and collects the optional technical exercises.

Component map

flowchart TB
    Browser[Browser on localhost] --> Gateway[Istio Gateway<br/>ports 80 + 443]
    Gateway --> OIDC[oauth2-proxy + Keycloak]
    Gateway --> Argo[Argo CD]
    Gateway --> Backstage[Backstage]
    Gateway --> Grafana[Grafana]
    Gateway --> Apps[demo + generated apps]
    Backstage --> Gitea[Gitea inside kind]
    Gitea --> Argo
    Argo --> Cluster[Kubernetes resources]
    Prom[Prometheus] --> Cluster
    Grafana --> Prom
    Policy[Kyverno] --> Cluster
Component Explore behavior
Argo CD Reconciles the app-of-apps tree from the in-cluster Gitea repository
Backstage Creates a repository, platform PR, catalog entity, and SSO callback without an external token
Gitea Receives the current checkout and holds all disposable repositories
Grafana Reads local Prometheus metrics from the cluster and applications
Keycloak Provides one disposable demo / demo identity
Istio + Gateway API Routes browser services over HTTPS without port-forwarding
cert-manager Creates a local CA and wildcard certificate
Kyverno Runs the production policy set in Audit mode
longhorn StorageClass Uses kind's local-path provisioner under the production class name

How a generated app becomes an image

Explore includes a disposable CI supply chain so that the code in each generated repository is the code that actually runs.

flowchart TD
    F[Backstage form] --> R[Generated Gitea repository]
    R --> T[Gitea Actions: test]
    T --> B[Build repository source]
    B --> I[Push commit-SHA image<br/>to disposable registry]
    I --> V[Commit SHA tag to deploy/values.yaml]
    V --> P[Backstage opens platform PR]
    P --> M[Platform administrator merges PR]
    M --> A[Argo CD sync]
    A --> Pod[New application pod]

make try builds and loads the fixed app-demo image so the platform has something to show immediately. It also builds Backstage from the current checkout, then starts an Explore-only Gitea Actions runner and registry. Every repository created afterward runs tests, builds its own Python and React source, pushes an image tagged with the source commit SHA, and commits that tag to deploy/values.yaml. The tag commit contains [skip ci], preventing an identical recursive build.

The seeded copy of this repository is the one exception. Its workflows are written for GitHub's hosted runners, and Gitea reads .github/workflows too, so make try turns Actions off for that repository alone. Left on, the platform pull request you are asked to merge would arrive showing failed checks that nothing in this cluster could ever satisfy.

Changing source on main repeats that path and changes the Deployment pod template, so Kubernetes replaces the pod. Changing deploy/ does not: the workflow declares paths-ignore: ["deploy/**"], because the theme, the greeting and the replica count are read at runtime or applied by Argo CD. Editing them is a sync, not a build. That file's own comment has always said so; the filter is what made it true.

The runner uses privileged Docker-in-Docker. That is acceptable only because this cluster is single-user, bound to localhost and deleted as a unit. Bare metal uses GitHub Actions, GHCR and a multi-architecture build instead; the Explore image contains only the host architecture.

Identity and request routing

Workload Authentication path Reason
Demo and generated apps Gateway ext_authz → oauth2-proxy The application contains no authentication code
Argo CD, Backstage, Grafana Native OIDC client → Keycloak Each product has its own user and role model
Gitea Local demo / demo account The PR merge is a separate platform-administration action; this is not SSO

The Gateway admits an application route only from a namespace with the expected environment label. Check one route:

kubectl -n app-demo get httproute demo \
    -o jsonpath='{.status.parents[0].conditions[*].type}'

Expected result: Accepted ResolvedRefs.

Without a browser session, the same route redirects to Keycloak:

curl -sk -o /dev/null -w '%{http_code} -> %{redirect_url}\n' \
    https://demo.127-0-0-1.sslip.io

Expected result: a 302 redirect to https://auth.127-0-0-1.sslip.io/realms/kensan/....

Why the certificate warning is accurate

cert-manager creates two issuers. A bootstrap self-signed issuer creates the kensan-lab explore CA; that CA signs one wildcard certificate for the Gateway. TLS encryption is present, but no public authority has verified domain ownership, so browsers correctly refuse to trust it automatically.

flowchart LR
    Self[Self-signed bootstrap issuer] --> CA[Explore root CA]
    CA --> Wildcard[*.127-0-0-1.sslip.io certificate]
    Wildcard --> Gateway[Istio Gateway]
    Browser -. not trusted by default .-> CA

The walkthrough gives both honest choices: bypass the warning for a localhost-only demo, or export explore-ca-tls and deliberately add that CA to the browser or operating-system trust store. make try never changes trust on the host.

Source change exercise

Scaffolding is not the only delivery path the platform owns; the ordinary one is a commit. In the repository the Golden Path created, open frontend/src/App.tsx, change one visible sentence with Gitea's edit button and commit to main.

A second Actions run tests and builds the source, pushes an image tagged with the new commit SHA, and writes that tag into deploy/values.yaml. Argo CD sees the tag change and replaces the pod; refresh the page to read the new sentence.

The interesting half is the failure. A commit that breaks the tests never reaches the image tag, so deploy/values.yaml keeps pointing at the last image that passed and the running pod is never disturbed. Nothing rolls back, because nothing rolled forward.

GitOps self-healing exercise

Scale the original demo by hand:

kubectl -n app-demo scale deploy demo --replicas=3
kubectl -n app-demo get deploy demo -w

Argo CD returns it to the Git-declared single replica within a second or two: the controller watches the cluster and does not wait for a Git poll to notice. A direct cluster edit is only a temporary opinion when selfHeal is enabled.

This is usually over before the next scrape, so the Grafana walkthrough drives CPU with the application's own load button instead of pretending the replica jump will always be graphed.

Application metrics

The Explore App Runtime dashboard combines:

  • container CPU from kubelet/cAdvisor;
  • desired and available replicas from kube-state-metrics;
  • http_requests_total from the FastAPI service;
  • p95 latency from http_request_duration_seconds.

The built-in demo and generated applications each have a ServiceMonitor for /metrics, scraped every 15 seconds. metrics-server is not installed, so kubectl top is unavailable; that does not affect Prometheus or Grafana.

The CPU panel's resolution is not the Prometheus scrape interval, which is a thing worth knowing before tuning the wrong knob. The kubelet's cAdvisor endpoint is scraped every 10 seconds, but cAdvisor timestamps its own samples and only recomputes per-container statistics about every 15 seconds, so the series advances at that rate however often Prometheus asks. Measured on a running Explore cluster: up for that endpoint records 6 samples a minute, container_cpu_usage_seconds_total records 4.

Policy exercise

All production ClusterPolicy objects run in Audit mode. The generated apps pass because the platform chart supplies security context and resource requests. Create a deliberately non-compliant pod:

kubectl -n app-demo run oops --image=nginx:latest --restart=Never
sleep 70
kubectl -n app-demo get policyreport -o json \
    | jq -r '.items[].results[] | select(.result=="fail") | "\(.policy): \(.message)"'
kubectl -n app-demo delete pod oops

You should see the mutable :latest tag and missing resource requests reported. Audit records violations without blocking the pod. The one-minute wait is Explore-specific; bare metal scans less frequently to avoid unnecessary etcd writes on constrained hardware.

Storage substitution

Production workloads request storageClassName: longhorn. Explore provides the same class name through kind's local-path provisioner so manifests bind without being rewritten:

kubectl get storageclass

The output should show both longhorn (default) and standard using the rancher.io/local-path provisioner.

This demonstrates the contract, not Longhorn's behavior. There is no replica rebuild, snapshot, backup, expansion, or node failover in a single-node cluster.

CNI chaining

Istio CNI appends itself to kindnet rather than replacing the cluster network:

docker exec kensan-lab-explore-control-plane \
    cat /etc/cni/net.d/10-kindnet.conflist \
    | jq -r '[.plugins[].type] | join(" ")'

Expected result: ptp portmap istio-cni.

On bare metal, the same setting chains Istio onto Cilium. Explore does not attempt to emulate Cilium L2 announcements on Docker's bridge network.

What kind cannot demonstrate

Capability Why it stays on bare metal
Cilium L2 load-balancer failover kind nodes are containers on one Docker bridge, not peers on a physical LAN
Longhorn recovery and node drain one node cannot lose or rebuild a replica
Multi-architecture scheduling every kind node has the host architecture
Vault root of trust unseal material and production credentials cannot be transferred to a visitor
Publicly trusted TLS the visitor does not own the production domain or its DNS credentials
Hardware telemetry temperature, Wi-Fi, disk, and router probes do not exist inside the Docker node
Production image supply chain Explore has a local runner and registry, but not GHCR credentials, release retention, signing, or multi-architecture builds

The bare-metal architecture pages describe these capabilities. The bootstrap guide is a reference for the live cluster, not a clean-room-tested installer; automation is planned.

Troubleshooting

A pod is Pending or the node is under memory pressure

Give Docker at least 8 GiB, then run make explore-down && make try.

Port 80 or 443 is already in use

Use sudo lsof -nP -iTCP:80 -sTCP:LISTEN and the equivalent command for 443. A previous Explore cluster is the usual cause.

An Argo CD Application does not become healthy

kubectl -n argocd describe application <name>

Sync waves order Application creation, not every dependency's readiness. Argo CD retries while charts and webhooks converge.

The browser warns about privacy

That is the expected local CA. Follow Accept the local certificate to bypass it or trust the generated root explicitly.

Gitea says you cannot merge the pull request

That is usually the signed-out view. Sign in with demo / demo. This local Gitea administrator owns the repository and can merge.

Everything is healthy but a browser URL does not resolve

Some corporate resolvers block wildcard DNS services:

for h in argocd backstage grafana demo auth gitea app2; do \
    echo "127.0.0.1 $h.127-0-0-1.sslip.io"; done | sudo tee -a /etc/hosts

Sign-in ends at /oauth2/callback with 403

The CSRF cookie lasts fifteen minutes. Restart in a private window and complete the login without leaving the form open.

kind cannot pull or parse the node image

Upgrade kind. The cluster pins a Kubernetes node image by digest, and v0.32.0 is the tested kind version.