DevOps deep dive | Live since August 20, 2026

GitOps delivery platform

The delivery infrastructure behind the Backstage refresh: a fully automated path from pull request to production where CI's only side effect is an image push, secrets never exist outside a vault, and every deploy is an auditable git commit. Designed, built, and taken live in three days on a shared AKS cluster — without a single ops ticket on the critical path — and since extended to an eighth repository without changing the model.

0manual steps per deploy
0plaintext secrets in git or CI
0ops tickets on the critical path
3days from empty namespace to live
8repos on in-cluster CI runners
71recorded architecture decisions

The delivery loop

One loop serves every kind of change — application code, plugin version bumps, configuration, and the Kubernetes manifests themselves. Nothing and nobody applies manifests or pushes images to the cluster by hand; merging to master is the deploy button, and a revert is the rollback button.

Delivery sequence: a developer change is raised as a pull request, tested and built by an in-cluster Docker-in-Docker runner, merged to master, pushed to the container registry, picked up by Flux image automation, committed back to master as a tag bump, and reconciled into a readiness-gated rolling update in the backstage-refresh namespace. Dev Pull request In-cluster runner (dind) Container registry Flux (image automation) master backstage-refresh namespace change + local CI gauntlet tests ×2 node versions + image assembles green → squash merge master push docker push master-<timestamp>-<sha> scan (2 min) commit tag bump via deploy key [skip ci] reconcile overlay rolling update, readiness-gated
Sortable image tagsmaster-<14-digit-UTC-timestamp>-<short-sha> — Flux's ImagePolicy picks the highest tag numerically; no latest-tag ambiguity, and any historical tag redeploys by reverting one commit.
Config deploys without imagesThe app-config ConfigMap is hash-suffixed by kustomize, so a config-only merge rolls pods in minutes with no rebuild.
One writer to the clusterOnly Flux applies manifests. CI has no cluster credentials at all — a smaller blast radius than the legacy pipeline it replaced, which rendered manifests and secrets in CI and pushed them to a deploy branch.
Proven on day oneThe first production deploy ran the entire loop hands-off: in-cluster Docker-in-Docker build, registry push, automation commit, rollout to a healthy pod.

Architecture

Architecture: the instance repository and the nine plugin repositories live on GitHub. Flux in flux-system reconciles three things in the backstage-refresh namespace — the application, the Verdaccio npm registry and the CI runners. The External Secrets Operator renders secrets from Azure Key Vault using workload identity. The runners push images to the container registry, which Flux also scans. GitHub flux-system namespace: backstage-refresh instance repo (app + k8s/) 9× plugin repos GitRepository + 3 Kustomizations + image automation Backstage + Postgres Verdaccio private npm registry ARC controller + 9 runner scale sets (dind, custom image) External Secrets Operator objects Azure Key Vault (self-provisioned) Self-service ACR tag-bump commits npm publish yarn install docker push scans workload identity renders secrets

Three Flux Kustomizations split the estate by blast radius: the registry (which owns the namespace and the secrets plumbing, and deploys first), the application, and the CI runners — each reconciles independently, so a broken runner chart can never block an app deploy. The platform is deliberately self-contained: its own Key Vault, its own container registry, its own npm registry, and its own CI compute, all inside one namespace on a shared cluster, coexisting with the legacy instance it replaces.

Secrets: vault-only, identity-based

No secret value exists in git, in manifests, or in CI-rendered output. External Secrets Operator syncs Kubernetes Secrets from Azure Key Vault using workload identity federation — a Kubernetes service-account token exchanged at Entra ID for a managed-identity token, so there is no client secret to leak or rotate anywhere in the chain. The vault, the user-assigned managed identity, and the federated credential were all provisioned self-service after discovering the shared ops vault was inaccessible — including working around missing role-assignment rights by running the vault in access-policy mode.

Everything is a referenceManifests contain Key Vault key names only; ESO renders real Secrets in-cluster, including a templated dockerconfigjson for image pulls.
Proven chainThe workload-identity exchange (issuer + subject + audience, exact-match) was verified end to end before the app ever booted — the registry's htpasswd secret was the canary.

CI where the network says it must be

A hard discovery shaped the whole CI design: neither ingress controller on the cluster is internet-facing — even the "public" class resolves public DNS to a private load-balancer IP. GitHub-hosted runners could therefore never reach the private registry. Rather than wait on a networking change, the platform runs GitHub Actions inside the cluster: actions-runner-controller with one ephemeral, scale-from-zero runner scale set per repository, a custom runner image with the CI toolchain baked in (built in-cluster with kaniko when the upstream image registry proved unreachable from workstations), and Docker-in-Docker for real image builds. A foreign cluster-wide runner controller that silently deleted our resources was diagnosed with a version-probe matrix and neutralized with a Kubernetes validating admission policy — the full story is in the case files.

The private registry as a product

Plugin distribution moved from vendored tarballs (fragile, checksum-pinned, fetch-script-dependent) to a Verdaccio private npm registry deployed by Flux. It is engineered as a deliberate stop-gap with an exit contract: the registry URL and credential exist in exactly one file per consumer, packages use plain semver, and the full release history was backfilled with integrity verification against the previously pinned hashes — so a future managed registry replaces it with configuration-only changes. Release immutability carries over as deny-unpublish configuration; publishing requires a dedicated CI identity; nothing is readable anonymously.

Empirical root-causingWhen token auth failed mysteriously, the fix came from reading the registry's authentication plugin source inside the shipped container image — not from guesswork. Auth moved to basic idents across all seven repositories the same evening.
Backfill, verifiedEvery previously released package version was republished byte-for-byte and checked against pinned sha512 integrity hashes before the old channel was retired.

Operating principles

Git is the only source of truthNo generated manifests, no deploy branches, no drift by construction.
Empiricism over folkloreEvery cluster assumption was probed before being relied on — API versions served, ingress reachability, controller behavior, auth formats.
Self-service over ticketsEach ops dependency discovered during bring-up was converted into something the team owns: vault, ACR, runners, registry.
Decisions are artifactsSeventy-one numbered decisions with dates, rationale, and amendments; every reversal traceable.
Documentation ships with the systemThe architecture and lifecycles live as a TechDocs page inside the portal itself, diagrams included.
Guardrails are codeRelease immutability, changeset-enforced SemVer, admission policies, and prune protection are all machine-enforced, not conventions.
Supporting sources: infrastructure-architecture TechDocs page; migration roundup decisions log (rounds 12-20) and work-remaining tracker through 09/04/2026; Aspirations-Backstage pull requests #8-#25