Debugging case files

SymptomSeven Helm releases installed cleanly, GitHub Actions jobs queued forever, and the AutoscalingRunnerSet custom resources simply did not exist — while Helm insisted they did. Our controller's logs showed it failing to add a finalizer to a resource that had vanished between two reconcile steps.

InvestigationRe-applied a resource by hand with a watch running (kubectl get -w --output-watch-events): ADDED, then DELETED, two seconds apart, no deletion timestamp phase — something with delete authority was killing it instantly. No owner references, so not garbage collection. The Helm chart's own discovery error had earlier revealed a second, cluster-wide actions-runner-controller installed by another team — and ARC deletes any runner set whose version label doesn't match its own build. Ours was 0.14.2; theirs was unknown and unreadable (no cross-namespace access).

Root causeTwo ARC controllers watching the same resources. The foreign one enforced version-mismatch cleanup on everything cluster-wide, deleting our resources seconds after creation — a mutual-destruction design where only one controller's version can survive.

FixScaled our controller to zero, then applied thirteen probe resources labeled with every plausible ARC version — the sole survivor identified the foreign controller as 0.13.0. Rather than downgrade onto an unknown team's controller, wrote a ValidatingAdmissionPolicy denying deletion of our runner sets by any service account outside our namespace, restored our controller, and re-created the resources. They survived; jobs ran within minutes.

LessonWhen something disappears in Kubernetes, watch it die before theorizing — the event stream distinguishes GC, finalizers, and direct deletes in one look. And a probe matrix turns "we can't see the other tenant's config" into data.

Symptomnpm login against the new registry returned "user registration disabled" — for a user that existed, with a password that verified. Registration being disabled was intentional; logging in should not be registration.

InvestigationEliminated layers one at a time: the password verified against the stored bcrypt hash with three independent implementations (apache htpasswd, bcryptjs v2, bcryptjs v3 — killing a promising wrong theory about the $2y$ hash prefix); the secret mount, config path, and file contents were confirmed correct; the failure reproduced locally in Docker with identical config — so it was software behavior, not the cluster. Debug logging showed the request never reached authentication at all. Final step: read the authentication plugin's source inside the shipped container image.

Root causeVerdaccio 6 routes the login endpoint through its add-user path, whose sanity check rejects with "registration disabled" (max_users: -1) before the existing-user branch — and the core has no fallback to authenticate. With registration disabled, token minting is structurally impossible; even a correct login would return "username already registered."

FixDropped tokens entirely: HTTP basic-auth idents, which hit the working authenticate() path on every request. Verified against the live registry (authenticated whoami, 404-not-401 on scoped reads, 401 anonymous), then rolled the credential format through all seven repositories' CI and local tooling the same evening.

LessonWhen behavior contradicts documentation, the vendored source in the running image is the ground truth — and it's always one docker run --entrypoint sh away. Also: disprove your favorite theory first; the bcrypt rabbit hole was three one-minute experiments instead of an afternoon.

SymptomFlux's first reconcile of the registry failed: a manifest referenced by the kustomization did not exist on master — yet it sat right there in the working tree, and git status showed a clean repo.

Investigationgit ls-tree proved the file existed on no branch, ever. git check-ignore -v named the culprit in one command: a broad *secret* ignore pattern, added long ago for safety, silently matched externalsecret-htpasswd.yaml — a manifest that contains only vault references. A sibling file had survived only because someone once force-added it.

Root causeSafety-net ignore patterns collide with declarative infrastructure naming: External Secrets manifests are supposed to have "secret" in their names precisely because they contain none.

FixCommitted the file with a scoped carve-out — then, when a second directory hit the identical trap a day later (externalsecret-gha-runner-app.yaml, breaking the CI-runner rollout), replaced per-directory exceptions with a whole-tree !k8s/** carve-out and a comment explaining the incident so it cannot recur.

LessonAn ignored file is invisible in every tool that matters — status, diff, PR review — which makes gitignore bugs a class of silent data loss. When the same trap fires twice, the fix is structural, not another exception.

SymptomEvery docker build and docker pull against ghcr.io from the workstation died with connection resets — consistently, across retries and hours, while other registries worked fine.

InvestigationThe corporate proxy was resetting ghcr.io specifically; no local fix available. But the cluster's nodes pulled from ghcr.io routinely — the ARC controller images came from there minutes earlier. The build didn't need the workstation at all.

Root causeAsymmetric egress: workstation traffic traverses an interfering proxy; cluster egress does not.

FixMoved the build to where the network works: a one-shot kaniko Job in the namespace — Dockerfile via ConfigMap, push credentials via the ExternalSecret-rendered dockerconfigjson already in place for image pulls. Base pull, build, and push to our ACR completed in about seventy seconds. The pattern is now the documented way to rebuild the runner image.

Lesson"The network is broken" is usually "this vantage point's network is broken." Clusters are excellent build machines you already have — and the secret plumbing you built for runtime often solves build-time auth for free.

SymptomCI failed four different ways in sequence as jobs moved onto the stock minimal runner image: yarn missing, then wget: command not found mid-action, then native modules dying with "Unable to detect compiler type," then — with tooling fixed — five React tests flaking on jest's five-second timeout.

InvestigationEach failure was localized from the runner-side logs rather than rerun-and-pray: corepack solves yarn (bundled with node), the typos action shells out to wget, node-gyp needs a compiler toolchain the image doesn't carry, and the timeouts correlated with node CPU contention while multiple runner pods shared a node — including one run that raced the image rollout and executed on an old-image pod, which the pod-age timeline exposed.

Root causeMinimal runner images fail in layers: every job class (actions, native builds, heavy test suites) peels back a new missing assumption that GitHub-hosted images had silently satisfied.

FixA custom runner image with the toolchain baked in (wget, jq, build-essential, python3, git-lfs) built in-cluster — plus corepack enablement in workflows, and a policy for the tail: a test that flakes twice gets its timeout raised in code, not another rerun. The instance's full gauntlet — two node versions plus a Docker-in-Docker image build — went green the same night and deployed production.

LessonMigrating CI compute is a dependency audit you didn't know you owed. Bake the image once instead of apt-getting in every job, and treat "it passed on rerun" as a bug report, not a resolution.