Kubernetes · Networking · 2026-09-03 · 5 min read

HA egress on Cilium OSS without paying for Enterprise

An on-prem Kubernetes cluster (kubeadm on VMware) needed a fixed egress IP: third parties whitelist one address, and all outbound traffic to them must originate from it. Cilium's egress gateway does exactly that — it steers matching traffic through one gateway node, SNATed to the designated IP.

The problem is the word one. In the Cilium this cluster runs — 1.17 OSS — the egress gateway does no failover. We verified it in testing: with several candidate nodes labeled, the datapath picks one deterministically, and when that node dies, matching traffic is dropped until someone intervenes. Gateway HA with leader election ships in Isovalent's enterprise egress gateway; the open-source version leaves the seat empty. So your carefully whitelisted egress IP is now a single point of failure with a pager attached.

The design: float the IP, follow the leader

Two moving parts, each doing the thing it's good at:

kube-vip floats the egress IP itself. It runs on the candidate worker nodes, elects a leader through a Kubernetes lease, and the winner announces the VIP with gratuitous ARP. Node dies → lease expires → another node takes the IP and GARPs the network. This is battle-tested plumbing; the same mechanism many clusters already use for their control-plane VIP.

A small label-sync controller follows the lease. Cilium doesn't care where the VIP lives — it routes egress traffic to whichever node carries the gateway label. So the missing piece is embarrassingly specific: keep exactly one node labeled, and make it the node that currently holds the kube-vip lease. The controller watches the lease and patches node labels accordingly. That's the entire job description.

kube-vip alone doesn't solve it (the label would still point at the dead node) and relabeling alone doesn't solve it (the IP wouldn't move). Together they give OSS Cilium what Enterprise sells: egress that survives a node failure.

From PoC to something you can page people about

The proof of concept was a bash script running in a tmux session — it validated the approach and would have died with my SSH connection. The production version is a Go operator on controller-runtime:

• Two replicas with leader election and pod anti-affinity, so the replicas live on different nodes and a node failure can never take both.
• Minimal RBAC: leases get/list/watch, nodes get/list/watch/patch. Nothing else.
• Hardened container: non-root, read-only root filesystem, no capabilities.

The interesting part is the invariants, because label-flipping has sharp edges:

• Add before remove. The new holder gets the label before the old one loses it. A few milliseconds with two labeled nodes is harmless; a window with zero labeled nodes is an egress outage. Order of operations is the difference.
• No leader → touch nothing. If the lease is momentarily holder-less mid-transition, the controller freezes rather than deleting the existing label. Stale-but-working beats clean-but-broken.
• Level-triggered, not edge-triggered. Every event triggers a full recomputation of desired state, plus a resync every ≤60 s even if nothing happens. Manual label drift heals itself.

The cold-start scare

One observation from the PoC nearly killed the whole design: the first failover onto a node that had never held the egress IP took over 30 minutes to converge — the Cilium agent was slow to recognize the egress IP as local. If that reproduced, a real failover landing on a "virgin" node would mean a half-hour egress outage, and the design would be worthless for production.

This became the gating spike, ahead of all the packaging work: reproduce it with a genuinely clean node, or bury it. Result: it didn't reproduce — a virgin node converged in ~5 seconds, consistently. Whatever produced the 30-minute convergence was an artifact of the PoC environment, not a property of the design.

Two lessons folded into one: chase your single worst observation before building anything on top of the design it questions — and remember that a proof-of-concept environment is itself a prime suspect. We could have shipped pre-warming machinery (rotating the VIP across all candidates at setup) to mitigate a problem that didn't exist.

Failure modes, stated out loud

Writing these down explicitly was worth more than any of the code:

Four Prometheus alerts cover the system: lease holder ≠ labeled node, zero-or-two labeled nodes, controller down, lease without holder. Each one maps to a row in that table.

Results

Failover measured end-to-end — from killing the gateway node to egress traffic leaving with the whitelisted source IP again, verified with packet capture on the new gateway:

node C → node B:  5 s
node B → node C:  4 s

For short-lived HTTP traffic, in-flight connections just retry and nobody notices. The VMware side needed its own validation (the port group must accept MAC changes / forged transmits for GARP-based IP migration to work), which an existing keepalived control-plane VIP had conveniently already proven. And the two planes stay isolated: the worst possible kube-vip failure returns egress to its old static-IP behavior — the API server VIP, owned by keepalived on the control-plane nodes, is never in the blast radius.

Takeaways

• 1. When the vendor gates HA behind Enterprise, check how small the missing piece really is. Here it was a lease-watching label syncer — a weekend of controller-runtime, not a re-architecture.
• 2. Invariants first, code second. Add-before-remove and no-leader-no-touch are what make the controller safe to run unattended; they were designed before the Go rewrite, on the whiteboard.
• 3. Gate the design on its scariest observation. A >30-minute cold start would have been disqualifying. It earned a focused spike — and turned out to be PoC noise.
• 4. State your failure modes in a table. "Both replicas die and egress keeps working" is a sentence you want written down and alarmed before it happens at 3 AM.