blog
War stories from production — real incidents, real configs, measured results.
Observability · 2026-09-04 · 5 min
The drop rule that ate the error logs
"Add an alert for failed flows" sounded like a one-day task on a FlowFuse/Node-RED platform. Underneath it: logs that didn't exist, an unanchored Alloy drop stage silently discarding error lines, restarts that GitOps couldn't see, and a chronic failure nobody was watching. Alerting was the easy part.
Kubernetes · Networking · 2026-09-03 · 5 min
HA egress on Cilium OSS without paying for Enterprise
Cilium's open-source egress gateway pins your fixed egress IP to one node — and does no failover when that node dies. How kube-vip plus a ~200-line controller-runtime operator got us 4–5 second egress failover on an on-prem cluster, and the >30-minute cold-start scare that almost killed the design.
Homelab · Security · 2026-09-03 · 4 min
When the office router became a weapon
Anatomy of a live DNS amplification attack — how a 24/7 Grafana stack on a Raspberry Pi caught an office MikroTik being abused as a ~37× reflector, and why the fix belongs in the raw table, not the filter chain.
AWS · Troubleshooting · 2026-09-03 · 5 min
A 599 MiB image, 91 minutes, and the region next door
Sustained TCP transfers from an on-prem datacenter to ECR in one AWS region died mid-flight at ~2.5 MB, turning image pulls into 91-minute ordeals. Root cause was outside our perimeter — so we moved the endpoint instead, and hit every cross-region ECR trap on the way.
AWS · Databases · 2026-09-03 · 5 min
The typo that rehearsed a production migration
A dev Aurora cluster named "stagging" needed renaming anyway — so we used it to rehearse a production MySQL migration end to end. Docker lab first, then a full dress rehearsal with 21 real services, a 15-minute cutover, 772 tables verified by exact count, and a failure-mode catalog you can only build by breaking things on purpose.