Proxmox ZFS Replication and HA: What a Node Failure Loses

When a Proxmox node dies and HA restarts its guest from a ZFS replica, the guest comes back on the surviving node exactly as its disk stood at the last replication snapshot. The writes made after that snapshot are not gone yet. If the dead node's disk survived, they are sitting on it. They go when that node comes back and Proxmox ZFS replication, which now runs the other way, reaches it: the receiving side's zfs recv -F rolls the returning disk back to the snapshot both copies share before it applies anything.

We followed one set of writes through all of it on Proxmox VE 9.2. A container appended a numbered line to a file every second on a three-node cluster, replicating to a partner node every minute with HA watching, and node A lost power 40 seconds after the last snapshot. In those 40 seconds it wrote lines 126 to 164. This post is where those 39 lines went.

The cluster we cut the power on

Three Proxmox VE 9.2 nodes (pve-manager 9.2.25, kernel 7.0.14, OpenZFS 2.4.4), built as VMs on our lab cluster, one ZFS pool each. Two Alpine containers on node A write a timestamped line to /root/counter.log and call sync every second. Both replicate to node B every minute, the shortest schedule Proxmox accepts; the default is every 15 minutes. Both are HA resources. Container 100 carries a strict node-affinity rule naming A and B. Container 101 carries none.

Containers kept the lab small, and on our reading of the code their disks go through the same replication path as a VM's. At 11:10:40 we hard-stopped node A from the host, 40 seconds after the 11:10:00 replication.

How much a Proxmox HA failover loses with ZFS replication

Everything written since the last replication snapshot. Container 100 came back on node B with this in its file:

1793876999. 124
1793877000. 125
BOOT 1793877174.
1793877174. 127

Line 125 is 11:10:00, the snapshot. The next line is the container booting on B at 11:12:54. Lines 126 to 164 never reached B, which is the trade the Proxmox storage replication docs warn about in one sentence: "some data loss between the last synced time and the time a node failed". On the default schedule that window is up to 15 minutes plus the run itself.

The restart itself took about the two minutes the HA manager docs call typical. Seen from node B, counting from the power cut:

+10.6 s    node zrep-a unknown
+61.0 s    zrep-a fence, both containers fence
+130.8 s   HA recovers ct:100 to zrep-b and ct:101 to zrep-c
+134 s     first line written by ct:100 on zrep-b

Most of that is the cluster making sure node A is really dead before it starts anything elsewhere, which is how Proxmox HA fences a node. The gap in the data is what ZFS replication added. Two more problems arrived in the next minute and the next quarter of an hour.

HA sent the other container to the node without its disk

Container 101 had no rule, and HA recovered it to node C, which holds no copy of it:

11:12:50 recover service 'ct:101' from fenced node 'zrep-a' to node 'zrep-c'
11:12:52 zfs error: cannot open 'tank/subvol-101-disk-0': dataset does not exist
11:13:10 relocate service 'ct:101' to node 'zrep-b'
11:13:13 migration aborted
11:13:40 recovery policy for service ct:101 failed, entering error state. Failed nodes: zrep-c, zrep-c

HA chooses a recovery node from the affinity rules, then the scheduler's score, then the node name. Nothing in that function (select_service_node in pve-ha-manager 5.2.5) reads a replication job. Container 100 went to B because its rule allowed only B once A was dead. That left B running one guest, and the default scheduler counts guests per node, so on our reading of that function C scored better for 101. The relocation to B, where the replica sat, failed before moving a byte: migrating a guest to its replication target runs that job first, and the job failed on the same missing disk. Three minutes after the power cut the container was in the error state, which HA leaves alone until someone runs ha-manager set ct:101 --state disabled and moves it by hand.

The fix is one strict node-affinity rule per guest naming only the nodes that hold its replica (on Proxmox VE 9 these replaced HA groups, and the HA docs give the same advice for anything only some nodes have), or a replication job to every node HA is allowed to pick:

# ha-manager rules add node-affinity pin-101 --resources ct:101 --nodes zrep-a,zrep-b --strict 1

Without it, on the default basic scheduler, which node a guest lands on depends on how many guests each survivor happens to be running at that moment.

When the failed node comes back, replication deletes its newer data

Node A was off for 6 minutes 51 seconds. When it booted, its copy of container 100's disk still ended at line 164, and ZFS counted 60K written since the 11:10:00 snapshot. Those 39 lines existed in exactly one place.

Proxmox had already turned the replication job around. When a guest runs on the node its job was replicating to, the replication code makes that node the source and the old source the target, so job 100-0 now ran from B to A. B tried it at 11:13:01, about 10 seconds after HA restarted the container and while A was still dark, and failed. A node that has never run the job has no state for it (last_try starts at 0), so on our reading its first try is not held back by the schedule. After a failure the job waits 5 minutes from the start of the last try, then 10, then 15, then 30, and it is only retried while the target node is online (PVE/ReplicationState.pm). The docs mention only the 30.

11:13:01   B to A fails: No route to host
11:17:45   node A back in the quorum
11:18:01   retry, 300 s after the failed try (failed on our lab's address fault)
11:28:01   retry, 600 s later: incremental sync 11:10:00 => 11:28:01
11:28:22   successfully imported 'tank:subvol-100-disk-0'

After 11:28:22 node A's file held none of the 39 lines, and its line 126 read BOOT 1793877174., B's history written over A's. The receive runs zfs recv -F, which in the zfs-recv manual will "force a rollback of the file system to the most recent snapshot before performing the receive operation". The most recent snapshot the two copies shared was 11:10:00. The log calls it a success.

The overwrite we watched came on the second retry, because our node rejoined advertising the wrong address, a fault of the lab's cloud-init and not of Proxmox. The first retry fired 16 seconds after A was back in the quorum, because its due time, 11:18:01, was set by the failed try and not by A's return. That clock is the one to plan around: it starts at the survivor's first failed try, within a minute of the restart (10 seconds here), so on our reading of the code a node that stays down longer than about seven and a half minutes in a failover timed like ours (a little over two to that first try, five for the wait) comes back with its retry already due, and the scheduler runs it at its next pass.

So the order of work, on a cluster where those last writes could matter (a database, a mail queue, anything a person typed):

1. Give every HA guest on ZFS a strict node-affinity rule that names only nodes holding its replica, and set each job's schedule to the loss you would sign off on, not the 15-minute default.
2. Watch pvesr status for a fail count above zero. A job that stopped syncing last week makes the next failover lose a week.
3. When a node dies, before you power it back on, run pvesr disable <job> for each guest it held, the documented way to stop a job. On our reading of the code the scheduler skips a disabled job, but migrating the guest back to that node still runs it, so leave the guest where it is. If its node-affinity rule ranks the returning node higher, HA moves the guest back by itself when that node rejoins (failback is on by default), so turn that off first with ha-manager set ct:100 --failback 0. Bring the node back, copy whatever you need off its copy of the disk, decide which copy wins, then pvesr enable <job>. We did not run this rescue, so test it on your own cluster before you need it.

Do not apply any of this to a guest that cannot lose a second. ZFS replication is the wrong storage for it. Ceph lets the surviving node start the guest on the same disk, so there is no older copy waiting to be reconciled. The internet provider's cloud in our service-provider case study runs both: ZFS replication under the cheap single VPS, Ceph under the customers running their own billing. And if losing the last interval is fine and nobody would ever go looking for those writes (a web front end, a build runner), the rule in step 1 is the whole job and you do not need us for it.

For everything in between, paste the output of pvesr list, ha-manager config and cat /etc/pve/ha/rules.cfg (or your HA groups on Proxmox VE 8) into the "Anything we should know?" box on the Proxmox scoping form, with "Already on Proxmox, need help" and "ZFS local with replication" selected, or send it on WhatsApp at +254 713 403 044. You get every HA guest marked: starts on its replica, can land on a node without its disk, has no replica, or sits on shared storage, with the most it can lose at its current schedule, the rules to add, and a power-cut test scoped for the guest you care about most. Reading your output costs nothing, and the test is quoted only if you want it run.