AWS · Databases · 2026-09-03 · 5 min read

The typo that rehearsed a production migration

A client's production MySQL needed to move — new cluster, staged service-by-service cutover, no big bang. Production database migrations are the kind of change where the plan is worth exactly as much as the parts of it you've rehearsed. So we rehearsed all of it, at three levels of increasing fidelity, before touching production.

The third level had a gift attached: the shared dev Aurora cluster had been created, long ago, with a typo in its name — stagging. Renaming an Aurora cluster properly means creating a new one and moving everything over… which is exactly a migration. So the typo fix became the dress rehearsal: same procedure, same tooling, real applications, real data, and a useful outcome even if the rehearsal was the only thing it achieved.

Level 1: a Docker lab that breaks things on purpose

The migration design is binlog replication with GTID auto-positioning, plus replication filters to cut over schema by schema: once a schema's service moves to the new cluster, a filter shields it from residual writes still arriving from the old side. An earlier end-to-end rehearsal had validated the replication channel itself (encrypted cross-account snapshot, GTID auto-position, CDC with zero lag) — but not the filters. And the filters are what make a staged migration viable at all.

Two MySQL 8.0.46 containers (GTID on, binlog ROW), zero client infrastructure, zero cost, 30–45 minutes per run. What came out of it:

• replicate-wild-ignore-table, not replicate-ignore-db. The db-level filter has a documented evaluation subtlety (MySQL replication rules): a qualified ALTER TABLE schema.table executed from another schema's context still applies to the schema you believed protected. The wildcard table filter doesn't have that hole. We reproduced both behaviors live.
• The filter goes in after draining, never before. Applying it while transactions are still in flight silently and permanently discards the schema's last writes — their GTIDs get marked as executed, so nothing ever retries them. Order of operations, again.
• Filtered transactions arrive as empty transactions, so GTID consistency and auto-positioning survive the filtering. The channel stays healthy for every schema that hasn't been cut over yet.
• The filter value is a replacement, not an accumulation. Every cutover must enumerate all schemas already migrated, or the previous one loses its shield.
• Trust SHOW REPLICA STATUS, not performance_schema — the latter reported inconsistent filter rules during the lab.

The lab also reproduces the failure modes deliberately, with their recovery: an unfiltered source write silently overwriting a destination row (no error, no alarm — the scariest one); error 1032 when the destination deleted the row; 1062 on a primary-key collision — both of which stop the applier for every schema, including the unmigrated ones; and 1396 when both sides create the same user (CREATE USER IF NOT EXISTS is the vaccine). Six deliberate-error exercises made it into the runbook, each with the real expected output of every command. If the person running the day-D procedure has already seen error 1062 and fixed it, it's not an incident anymore — it's a checklist item.

Level 2: the questions only Aurora can answer

A Docker lab can't tell you how Aurora's parameter groups behave. Two things stayed open, and either could change the day-D procedure: does replicate-wild-ignore-table with ApplyMethod=immediate take effect without a reboot (Aurora replication filters)? And does the parameter group validate the filter syntax the way the engine does over SQL?

That's what the dress rehearsal was for.

Level 3: the dress rehearsal (thanks, typo)

The full production procedure, run against the misnamed dev cluster as source and a correctly-named new cluster as destination — with the ~20 real services that use it staying connected throughout.

Preparing the source. Binlog ROW plus the online GTID ladder (enforce_gtid_consistency=ON → gtid_mode=ON_PERMISSIVE → ON, as documented by MySQL) — with applications connected. Four restarts, measured: 10 s, 9 s, 8 s, ~8 s; every app reconnected on its own. Now the production plan says "four restarts of under ten seconds each", with evidence, instead of "some restarts".

Destination and replication. New cluster created via Terragrunt from the post-GTID snapshot (restore ~25 min); binlog replication with auto-position engaged, lag 0 sustained for the whole coexistence period.

The Aurora answers. ApplyMethod=immediate on the filter parameter takes effect without reboot — the cluster never left in-sync. Bonus finding: RDS merges its internal mysql.% filter with yours rather than replacing it. And the shield held on real Aurora: with the filter active, the source couldn't overwrite destination rows no matter what we threw at it.

The gotcha nobody had on the list. The destination's security group had no egress rule — Terraform removes the default-allow egress unless you re-declare it — and the replication IO thread just sat in Connecting, with no visible error anywhere. A silent, indefinite hang as the only symptom of a one-line SG omission. That's now a pre-flight check in the production runbook.

The cutover: 15 minutes end to end.

freeze 21 ECS services (desired=0)          ~2 min
drain: WAIT_FOR_EXECUTED_GTID_SET           = 0 instantly
integrity: 772 tables / ~93 GiB
  exact row counts on both sides, parallel  ~5.5 min → diff identical
repoint: 19 Secrets Manager secrets         58 s  (rollback: AWSPREVIOUS)
services back up and stable                 2 min 32 s
definitive multi-schema filter, no reboot   37 s

Connections verified on the destination, zero on the source.

Teardown that finishes the job. Replication channel stopped and reset (destination standalone), final snapshot taken, and the misnamed cluster destroyed via Terragrunt — seven resources, billing stopped. A rehearsal that leaves the old cluster running "just in case" isn't finished; it's a cost leak with sentimental value.

What this bought

The production migration plan now contains zero un-rehearsed steps. Every duration in it — restarts, restore, drain, count-validation, secret rotation, service recovery — is a measurement from this run, not an estimate. And the scary parts (silent data overwrite, the filter-before-drain trap, the SG hang) have all already happened once, on hardware nobody would cry about, with the recovery written down.

Takeaways

• 1. Find a disposable vehicle for your dress rehearsal. A typo'd cluster name was a gift: the rehearsal produced permanent value (the rename) even beyond de-risking production.
• 2. Practice the failures, not just the procedure. Six deliberate-error exercises turned the runbook's scariest scenarios into things the operator has already fixed once.
• 3. Sequence beats configuration. Both nasty data-loss modes in this design come from ordering (filter before drain; unfiltered coexistence writes), not from any wrong parameter value.
• 4. Validate by counting, not by vibes. 772 tables compared by exact row count in parallel took 5.5 minutes — a trivially cheap price for saying "identical" with evidence.
• 5. Make every step reversible and every teardown total. Secrets roll back via AWSPREVIOUS, clusters die via IaC with a final snapshot, and nothing stays up "just in case".