Our health check moved the default route to the backup line in 3 seconds. The WireGuard tunnel to head office took 186 seconds to come back, which is longer than the outage lasted. Same router, same failure, different NAT rules.
CloudSpinx designs and manages office networks in Kenya, and we recommend a second ISP with automatic failover on every one we build. So the question we cared about was not whether the backup line takes over. It was what an ISP failover does to the traffic already running when the primary dies: a web request, a long TCP session, a UDP flow shaped like a phone's SIP keepalive and a site-to-site VPN, each checked once a second through six staged failures in our lab. Here is what dropped in each failover test, for how long, which setting decided it, and how to test your own.
The failure we staged, and why it is the hard one
One Ubuntu 26.04 VM held the whole network in Linux namespaces: an office PC, a router with two WAN ports and nftables NAT, two ISPs, and a server standing in for the internet and for head office. Then ISP 1 stopped forwarding anything, while its address and the router's port facing it stayed up.
That is the failure a dual-ISP router has to catch. The fiber is cut somewhere down the road, or the provider's core has a bad hour, and the cable between the provider's box and your router still shows link. A router that fails over on link state alone did nothing in our first run. Every probe failed for the whole 90 seconds. When ISP 1 came back, the long TCP session took another 18 seconds to answer. TCP doubles its wait after each unanswered retransmission (RFC 6298), so its next retry came well after the line was back.
So we gave the router a health check: a ping to a host beyond ISP 1 every second, three misses to move the default route to ISP 2, three replies to move it back. That is aggressive on purpose. MikroTik's check-gateway, by comparison, checks the route's gateway with a ping or ARP request every 10 seconds by default and marks it unreachable after two timeouts, and a provider router that keeps answering while its upstream is dead keeps that check green. Pinging a host past the provider, the fix in the failover that never failed over, closes that gap. Detection turned out to be the easy part.
Masquerade: back in 3 seconds, and TCP reset at both switches
With masquerade NAT on both lines, new connections, the SIP-style UDP flow and the tunnel all answered again 3 seconds after the first lost probe. Head office's WireGuard peer simply picked up the tunnel's new public address, ISP 2's.
That is the Linux kernel being helpful. A masqueraded connection remembers which interface it leaves by, and when a packet tries to leave by a different one, the kernel kills the connection's tracking entry, address translation and all, and drops that packet (nf_nat_oif_changed in the source). The next packet starts afresh on ISP 2's address.
The bill lands on any TCP connection that is sending when the route moves. The server receives a packet from an address it has no connection with and answers with a reset, so our long session died 6.7 seconds into the outage. Our probe reconnected over ISP 2 at once. ISP 1 came back, the route moved home at 182.6 seconds, and that new session was reset too. Behind a router like this one, anything that keeps a TCP connection busy is reset at failover and, once it reconnects, again at failback. Our probe stood in for an RDP session to a hosted desktop, a database connection or a large upload, and the second reset arrives just as everyone decides the problem is over.
Static NAT on a fixed IP: the VPN that stays on the dead line
MikroTik's NAT documentation says masquerade "was designed for specific use in situations when public IP can randomly change", and that with src-nat instead, "connection tracking entries remain and connections can simply resume after a link failure". On a business line with a fixed public address, that reads as a reason to write one source NAT rule per line instead. We did, and broke ISP 1 again.
New connections moved in 3 seconds, as before. Nothing else moved. The UDP flow and the tunnel kept their NAT entries, still translated to ISP 1's address, and every keepalive refreshed those entries before they could expire. The router sent their packets out through ISP 2 still stamped with ISP 1's address. Our ISP 2 dropped them as spoofed, and without that filter the replies would have gone back to ISP 1, which was dead. The UDP flow was down 182 seconds and the tunnel 186, against an outage of 180: both came back only when ISP 1 did. That tunnel could be the branch link to head office. We ran WireGuard only; an IPsec VPN started behind the router, such as one backing up a circuit to AWS, deserves the same test. The long TCP session sat frozen for 214 seconds and then carried on, which sounds better until you notice it resumed 35 seconds after the line was back.
Failback had its own victim. A flow that started during the outage was translated to ISP 2's address, and when the route moved home it left by ISP 1, which in our lab drops packets carrying another network's address, as a provider filtering spoofed sources does. It was still dead 70 seconds later when the run ended.
- answering
- not answering, or reset
One kernel check makes the difference. A masqueraded flow is killed when its exit interface changes, so its next packet starts over on the backup address; a statically translated flow keeps the old address for as long as its keepalives refresh it.
If your router runs source NAT on a fixed address, this is the case to rule out first. Send the NAT section of the config and we will tell you which of these two you have.
Flushing the dead line's connections fixes it
Deleting the connection-tracking entries for the line being left, at the moment of each switch, turned static NAT into the masquerade result: everything back in about 3 seconds, nothing stuck at failback, and TCP reset at both switches. On our router that was one command in the failover hook, conntrack -D --reply-dst with the address of the line being left.
pfSense has a similar switch for gateway failure and ships it off: "Do not kill states on gateway failure" is the default, and its own documentation says clearing states "can help redirect traffic for long-lived connections such as VoIP phone/trunk registrations to another WAN". Its selective option has a catch worth reading twice. It cannot kill states created by default gateway switching, because those record the gateway as 0.0.0.0, so it only works where a policy routing rule chose the line. Where the router offers it, turn the flush on deliberately, after deciding which sessions you would rather reset than freeze.
How to test ISP failover so the result means something
We also ran the easy version of the test: taking the router's own ISP 1 interface down. The kernel dropped the route and cleared that interface's masqueraded connections at once, and the UDP flow and the tunnel did not lose a single probe at the switch. The router is told about that failure directly. When the line dies beyond a provider box that keeps its port up, it gets no such signal, and a pass proves nothing about that case.
- passes
- what the real failure does
A was an administrative link down on the router, which clears masqueraded entries in the kernel. A physical cable pull is a different event and was not tested.
Break the path beyond the provider's box instead. Unplug the fiber or the radio feed going into the ISP's router and leave the cable from it to yours alone, or ask the provider to shut the service at their end. Keep a call up, the VPN to head office pinging, an RDP session open and a large download running, and time each one. Then restore the line and time them again, because failback is a second switch, and in our runs it cost a second reset or a dead flow.
An office where nobody holds a call or a tunnel open across an outage needs only the first half of this: a health check that pings past the provider, and one honest test. Do that yourself; you do not need us for it.
If the office runs a site-to-site VPN, desk phones on a SIP trunk, or anything a partner reaches through an allowlist of your fixed address, send us the router make and model, an export of its NAT and failover configuration, and which of those services has to survive a switch, through the network scoping form or on WhatsApp at +254 713 403 044. You get back what drops on a real outage and for roughly how long, and the configuration change that fixes it. If we end up managing the network, the links and tunnels are monitored and the whole thing is one monthly figure. The reply is free and binds you to nothing.