Neal Spangler

Work Software & web

Documented an inherited cluster and added deploy checks

I took over the production Kubernetes cluster without documentation. I inventoried it, compared the running resources with Git, and added staging defaults and a typed confirmation for production deploys.

Staging and production deploy checks Config lives in a Git repo without secrets. The staging deploy script has no path to production and staging is the default target. Production takes a separate deploy command and a typed confirmation phrase, and changes are rehearsed on staging first. Separate staging and production deploys Git repo no secrets Staging script no path to prod Staging default target Prod deploy its own command Typed phrase confirm by hand Production live sites rehearsed here first
Staging is the default. Production takes its own command and a typed phrase.
17production deployments inventoried
31live resources Git did not track
4resources drifted from Git

The problem

The company's websites ran on this cluster. Some of what ran in production existed only in the cluster, and some had drifted from Git, so a routine deploy could undo a change nobody remembered making.

What I did

  • Regained admin access and inventoried the cluster: 3 nodes, 13 namespaces and every deployment.
  • Compared running resources with Git to find untracked resources and configuration changes.
  • A staging deploy script with no code path to production, and kube contexts that point at staging by default.
  • A kustomize export of the cluster into Git, with secrets kept out of the repository.
  • A Traefik v3 ingress upgrade, tested on staging with a scripted rollback.

What I took from it

Comparing the cluster with Git showed which resources existed only in production. I needed that list before deploying, because the repository did not describe everything the websites depended on.

Software & web

Rehearsed a legacy site's move to PostgreSQL 16

Moved 3.5 million rows from MySQL and MongoDB to PostgreSQL 16 in a rehearsal, with zero load errors.

Software & web

Fixed catalog images and automated Kubernetes deploys

Matched artwork to products missing images, fixed bad photos and descriptions, and automated deploys to Kubernetes.
Privacy