Neal Spangler

Work Software & web

Rehearsed a legacy site's move to PostgreSQL 16

The old site ran on MySQL 5.6 and MongoDB 4.4, and the new one is built on PostgreSQL 16. I rehearsed the whole move in isolated containers from read-only copies, away from the live system.

Legacy databases to PostgreSQL 16 MySQL 5.6, 253 tables, loaded through pgloader as 3,562,317 rows, and MongoDB 4.4 converted to JSONB as 345,084 records, both into PostgreSQL 16 in an isolated container. The new application booted on the result, and a product count check found a 45% gap. Read-only copies, isolated containers MySQL 5.6 253 tables MongoDB 4.4 document store pgloader 3,562,317 rows JSONB import 345,084 records PostgreSQL 16 isolated container New app booted on the result count check found the 45% gap
Loaded both databases into PostgreSQL 16 and booted the new app.
3,562,317rows moved from MySQL
0load errors across 253 tables
345,084MongoDB records converted to JSONB
45%product shortfall traced to one value

The problem

Before scheduling a cutover, I needed to check that the data could move intact and that the new application would run on it. The MySQL and MongoDB data had to fit one PostgreSQL schema.

What I did

  • Loaded every MySQL table with pgloader and converted the MongoDB collections into PostgreSQL JSONB.
  • Booted the new application's container image on the result and let its schema migrations run.
  • Ran the product import and checked its count against the old catalog. It came up short.
  • Traced the gap to a company ID that identified the master catalog. The importer had been treating that value as unused data and discarding it.

What I took from it

The tables loaded without errors, but the product count came up short. Comparing it with the old catalog exposed a value the importer had discarded even though it identified the master catalog.

Software & web

Documented an inherited cluster and added deploy checks

Documented an inherited Kubernetes cluster and added staging defaults, deploy checks and production confirmation.

Software & web

Fixed catalog images and automated Kubernetes deploys

Matched artwork to products missing images, fixed bad photos and descriptions, and automated deploys to Kubernetes.
Privacy