Neal Spangler

Work Infrastructure

Monitoring and recovery for servers and production PCs

I set up scheduled checks for servers, switches, printers and production PCs. The jobs report changes, attempt specific repairs where allowed, and alert me when someone needs to intervene.

Smart plug
Smart plug Photo: Gregory Varnum, CC BY-SA 4.0
56scheduled jobs on one Linux workstation
5checks before a smart plug cuts power
22workstations checked for overdue reboots

The problem

Some faults went unnoticed or misdiagnosed for months. A managed switch kept passing traffic while its management interface was hung, and a Mac had no working wired connection. When a print workstation froze, someone on site had to notice and pull the power.

What I did

  • Systemd timers for health checks, self-repairs and backups, plus a weekly scan that flags new hosts and open ports.
  • Alerts on state changes, with messages queued during an internet outage and sent when the connection returns.
  • Smart-plug recovery for production PCs that hard-hang. The configured checks must pass before power is cut. Repeated failures stop automatic recovery and request a physical cord pull.
  • Added checks for recurring faults. When the business database's web API stopped after an automatic update, I added detection and recovery for that failure.
Checks before a frozen PC is power-cycled When a production PC stops answering, a first alert is sent. The smart plug cuts power only if five checks pass in order: auto-recovery is armed, the PC failed ping and SSH on two runs in a row, the plug is reachable, the plug is on and drawing over 8 watts, and the cooldown and daily cap allow it. If any check fails, the watcher only alerts. If the PC stays hung after retries, it asks for a physical cord pull. PC stops answering first alert sent 1 Armed kill switch is on 2 Still down ping and SSH, two runs 3 Plug online check plug state 4 Drawing power on and over 8 W 5 Under limits cooldown and daily cap Power-cycle off, then auto-boot Alert only no power cut any check fails still hung after retries: asks for a cord pull
A failed check alerts me. Recovery code checks the PC before cutting power.

What I took from it

I traced 619 restarts and 612 crashes on a print workstation to our own watchdog. I added limits and reason logs to recovery scripts so I can see whether the fix is causing trouble.

Automation & AI

AI agents with command limits and approval checks

Claude Code and Codex handle remote tasks with approval checks and command-blocking rules enforced outside the model.
Privacy