AWS · Troubleshooting · 2026-09-03 · 5 min read

A 599 MiB image, 91 minutes, and the region next door

A deploy on a client's on-prem Kubernetes cluster was stuck in ImagePullBackOff. The image — a 599 MiB Node.js build hosted in Amazon ECR — eventually pulled. It took 1 hour, 31 minutes and 30 seconds.

The pattern

Sustained TCP transfers from the datacenter's egress IP to ECR in us-east-2 were dying mid-flight after transferring between 1.3 and 3.1 MB, intermittently, with a failure rate between 40% and 83%. Reproducible from three separate internal network segments, on a server with no container runtime at all, with bare curl. The only common factor was the public egress IP.

The region gradient was the interesting part: us-east-2 failed 40–83% of the time, us-west-2 around 20%, and us-east-1 never failed. That shape rules out a policy targeting "AWS" as a whole, and reproduction from independent internal segments rules out a per-segment firewall rule. Everything pointed at transit degradation toward that specific region's prefixes — somewhere between the datacenter's edge and the carrier. In other words: a root cause we didn't own and couldn't fix.

Why big images are disproportionately punished

The cutoff sits around 2.5 MB per TCP session, and container images aren't transferred as one stream — they're transferred per layer. Layers under the threshold almost never fail. The 599 MiB image had 4 layers above it (223, 197, 149 and 29 MiB).

With roughly a 60% chance of a large layer completing on a given attempt, the probability of all four passing in one round is about 0.6⁴ ≈ 13%. containerd retries per layer, so the pull converges eventually — but each failure wastes up to 223 MiB of transfer and the backoff accumulates. That's the whole 91 minutes, no exotic failure mode required.

It also explains why the app team's Dockerfile fix — which cut the image from 599 to 279 MiB, a 53% reduction — didn't solve anything: it still left the same four layers above the threshold. Shrinking the total reduces the cost of each failure, not the number of dice rolls. What matters here isn't image size; it's the layer size distribution.

Can't fix the path? Move the endpoint

Since us-east-1 was clean, the workaround was to serve the image from there: ECR cross-region replication plus seeding. Each step had a trap:

• Replication doesn't backfill. It only copies images pushed after the rule exists (AWS docs), so the existing tags had to be seeded by hand.
• Seed with crane cp, not docker pull/push. crane copies blobs as-is and preserves digests; the Docker daemon recompresses layers and can change them, which would have broken the whole "same content, different region" premise of the test.
• Scope the replication filter to the full repository name. PREFIX_MATCH is a prefix: a shorter filter would have silently dragged two sibling repositories along.
• ECR auth tokens are per registry host. The cluster's pull secret only held the us-east-2 registry entry, so the first pull from us-east-1 failed with no basic auth credentials. The kubelet merges the auths of every secret in imagePullSecrets, so a second, region-specific secret on the one deployment solved it — without touching the cluster-wide secret that a CronJob distributes to ~60 namespaces. Blast radius matters more than elegance during an incident.

The measurement traps

Two things nearly produced false conclusions, and they're worth more than the fix itself:

containerd's cache makes cross-region pull tests lie. Because digests are preserved, a node that already has the layers from region A "pulls" from region B in under a second, transferring nothing. An early test reported 334 MB in 796 ms — 420 MB/s, physically impossible over that uplink. Before measuring: remove the image, confirm via the kubelet's .status.images that the node is actually cold, then pull. We forced scheduling onto the one node that had never held the image.

A 401 from the registry is invisible to CloudTrail. The failed us-east-1 pull never showed up as an API event — because an unauthenticated request to the registry's /v2/ endpoint never becomes an authenticated AWS API call. During diagnosis, "CloudTrail is empty" was briefly read as "nobody tried to pull", when it's exactly what a credential failure predicts. Know which failures your telemetry cannot see.

Bonus trap: iam simulate-principal-policy returned implicitDeny for every action on both principals — including actions explicitly allowed on Resource: "*". The tell was MatchedStatements: []: the simulator had loaded no policies at all (insufficient read permissions) and reports that as a deny rather than an error. It contradicted CloudTrail, which showed both identities actively pulling and pushing. When a simulator and reality disagree, believe reality.

Result

Same image, same digest, same node, real cold pull:

us-east-2:  1h 31m 30s   (~3.9 MB/s, 40–83% of sessions cut, many retries)
us-east-1:  1m 22s       (7.6 MB/s, zero retries)

67× faster. With the honest caveat that a single clean pull is n=1 — us-east-2 completed 6 of 10 attempts in the original measurements, so one good pull from there is possible too. But 82 seconds against 5,490 isn't variance; it's the difference between a clean pass and a pile of backoffs. A four-cell interleaved measurement harness (2 tags × 2 regions, classified by exit code) was left behind for anyone who wants confidence intervals.

And it's a workaround, not a fix: the CI runners still push through the degraded path, and the actual escalation — to the network team and the carrier that owns the egress prefix — is where the root cause lives.

Takeaways

• 1. Layer size distribution beats total image size. An image is as fragile as its largest layer. Multi-stage builds and splitting fat COPY layers do more for pull reliability than shaving total MiB.
• 2. When you don't own the root cause, change what you do own. We couldn't fix a carrier's transit; we could serve the bytes from a region the path to which worked.
• 3. Kill the cache before you measure. Digest-preserving copies make cache hits indistinguishable from miraculous throughput.
• 4. Map your telemetry's blind spots. Registry 401s don't reach CloudTrail; a broken policy simulator reports denies. Both will send you chasing ghosts.
• 5. Prefer the smallest blast radius under incident pressure. One extra secret in one namespace beat editing the secret that 60 namespaces depend on.