The delivery infrastructure behind the Backstage refresh: a fully automated path from pull request to production where CI's only side
effect is an image push, secrets never exist outside a vault, and every deploy is an auditable git commit. Designed, built, and taken live
in three days on a shared AKS cluster — without a single ops ticket on the critical path — and since extended to an eighth repository
without changing the model.
0manual steps per deploy
0plaintext secrets in git or CI
0ops tickets on the critical path
3days from empty namespace to live
8repos on in-cluster CI runners
71recorded architecture decisions
The delivery loop
One loop serves every kind of change — application code, plugin version bumps, configuration, and the Kubernetes manifests themselves.
Nothing and nobody applies manifests or pushes images to the cluster by hand; merging to master is the deploy button, and a revert
is the rollback button.
Sortable image tagsmaster-<14-digit-UTC-timestamp>-<short-sha> — Flux's ImagePolicy picks the highest
tag numerically; no latest-tag ambiguity, and any historical tag redeploys by reverting one commit.
Config deploys without imagesThe app-config ConfigMap is hash-suffixed by kustomize, so a config-only merge rolls pods in
minutes with no rebuild.
One writer to the clusterOnly Flux applies manifests. CI has no cluster credentials at all — a smaller blast radius than the
legacy pipeline it replaced, which rendered manifests and secrets in CI and pushed them to a deploy branch.
Proven on day oneThe first production deploy ran the entire loop hands-off: in-cluster Docker-in-Docker build, registry push,
automation commit, rollout to a healthy pod.
Architecture
Three Flux Kustomizations split the estate by blast radius: the registry (which owns the namespace and the secrets plumbing, and deploys
first), the application, and the CI runners — each reconciles independently, so a broken runner chart can never block an app deploy. The
platform is deliberately self-contained: its own Key Vault, its own container registry, its own npm registry, and its own CI
compute, all inside one namespace on a shared cluster, coexisting with the legacy instance it replaces.
Secrets: vault-only, identity-based
No secret value exists in git, in manifests, or in CI-rendered output. External Secrets Operator syncs Kubernetes Secrets from Azure Key
Vault using workload identity federation — a Kubernetes service-account token exchanged at Entra ID for a managed-identity token,
so there is no client secret to leak or rotate anywhere in the chain. The vault, the user-assigned managed identity, and the federated
credential were all provisioned self-service after discovering the shared ops vault was inaccessible — including working around missing
role-assignment rights by running the vault in access-policy mode.
Everything is a referenceManifests contain Key Vault key names only; ESO renders real Secrets in-cluster, including a templated
dockerconfigjson for image pulls.
Proven chainThe workload-identity exchange (issuer + subject + audience, exact-match) was verified end to end before the app
ever booted — the registry's htpasswd secret was the canary.
CI where the network says it must be
A hard discovery shaped the whole CI design: neither ingress controller on the cluster is internet-facing — even the "public" class
resolves public DNS to a private load-balancer IP. GitHub-hosted runners could therefore never reach the private registry. Rather than wait
on a networking change, the platform runs GitHub Actions inside the cluster: actions-runner-controller with one ephemeral,
scale-from-zero runner scale set per repository, a custom runner image with the CI toolchain baked in (built in-cluster with kaniko when
the upstream image registry proved unreachable from workstations), and Docker-in-Docker for real image builds. A foreign cluster-wide
runner controller that silently deleted our resources was diagnosed with a version-probe matrix and neutralized with a Kubernetes
validating admission policy — the full story is in the case files.
The private registry as a product
Plugin distribution moved from vendored tarballs (fragile, checksum-pinned, fetch-script-dependent) to a Verdaccio private npm registry
deployed by Flux. It is engineered as a deliberate stop-gap with an exit contract: the registry URL and credential exist in exactly
one file per consumer, packages use plain semver, and the full release history was backfilled with integrity verification against the
previously pinned hashes — so a future managed registry replaces it with configuration-only changes. Release immutability carries over as
deny-unpublish configuration; publishing requires a dedicated CI identity; nothing is readable anonymously.
Empirical root-causingWhen token auth failed mysteriously, the fix came from reading the registry's authentication plugin source
inside the shipped container image — not from guesswork. Auth moved to basic idents across all seven repositories the same evening.
Backfill, verifiedEvery previously released package version was republished byte-for-byte and checked against pinned sha512
integrity hashes before the old channel was retired.
Operating principles
Git is the only source of truthNo generated manifests, no deploy branches, no drift by construction.
Empiricism over folkloreEvery cluster assumption was probed before being relied on — API versions served, ingress reachability, controller behavior, auth formats.
Self-service over ticketsEach ops dependency discovered during bring-up was converted into something the team owns: vault, ACR, runners, registry.
Decisions are artifactsSeventy-one numbered decisions with dates, rationale, and amendments; every reversal traceable.
Documentation ships with the systemThe architecture and lifecycles live as a TechDocs page inside the portal itself, diagrams included.
Guardrails are codeRelease immutability, changeset-enforced SemVer, admission policies, and prune protection are all machine-enforced, not conventions.
Supporting sources:
infrastructure-architecture TechDocs page; migration roundup decisions log (rounds 12-20) and work-remaining tracker through 09/04/2026;
Aspirations-Backstage pull requests #8-#25