A pod stuck on Multi-Attach error for volume after its node lost power is waiting for that node to let go of the volume, and a node with no power never will. One command lets go on its behalf, and it is only safe once you know the node is off:
$ kubectl taint nodes w1 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
node/w1 taintedOn our lab cluster it got a Longhorn StatefulSet writing again 27 seconds later, after more than 19 minutes down, 13 of them stuck in Terminating with no replacement. That is the case Kubernetes documents the taint for. The pod that showed the attach error was our Deployment's replacement, freed another way, covered below. On Kubernetes 1.37, and from the 1.36.4 and 1.35.9 patch releases on, the event no longer says Multi-Attach error at all. It reads Waiting for detach for volume, and the fix is the same.
The cluster was two Debian 13 VMs on Proxmox running k3s v1.37.2, with Longhorn v1.13.0 keeping one replica of each volume on each node. Three single-replica workloads sat on the worker, each writing and syncing a numbered line every second: a StatefulSet on k3s's bundled local-path storage, a StatefulSet on Longhorn and a Deployment on Longhorn. We cut the worker's power with qm stop twice, first with Longhorn's defaults and then with its node-down setting changed. Which storage sits under a stateful pod decides what a dead node costs, and on your own hardware that choice is yours rather than a cloud provider's. CloudSpinx builds and runs Kubernetes on clients' own hardware in Kenya.
The policy and nobody-acts rows count from the eviction, which comes about 350 s after the power cut on default tolerations.
What the Multi-Attach error is waiting for
A ReadWriteOnce volume from a CSI driver that attaches, Longhorn in our case, is attached to one node at a time, and Kubernetes records that in a VolumeAttachment object. The attach-detach controller will not attach the volume anywhere else while that object names the old node, and it will not detach while it believes a pod there still uses the volume.
A power cut never changes that belief by itself. Our worker went NotReady at 50 seconds and the taint-based eviction marked all three pods for deletion at 350, the default 300-second toleration we timed in the node failure post. Then they sat in Terminating, because the only thing that can confirm a pod stopped is the kubelet, on a machine with no power. The Deployment got a replacement on the surviving node within a second. The StatefulSets got none, by design: two pods holding one identity is what a StatefulSet exists to prevent.
The event on the replacement tells you which of two stages you are in:
Warning FailedAttachVolume Waiting for detach for volume "pvc-1c4dee57-..." Volume is already used by pod(s) lhd-6b6b475cf8-k2rfb"Already used by pod(s)" names the old pod, which still exists and is the blocker. "Volume is already exclusively attached to one node" means the old pod is gone and the controller is waiting to detach. Both started with Multi-Attach error until Kubernetes PR #138837 renamed the prefix. The controller reports the event once per volume, so after you act the stale line stays the newest one. Watch kubectl get volumeattachments instead. Once the row naming the dead node is gone, the new attach follows within the controller's 60-second sync.
Does Kubernetes force-detach after six minutes?
It does, but the six minutes start when the old pod is deleted, not when the node dies. We left the Deployment's replacement alone for 412 seconds after the eviction. The VolumeAttachment never moved, and the controller logged "Waiting for detach: volume is in use by pods" once and nothing after it. In the v1.37.2 source the timer starts only once no pod wants the volume on that node, and a Terminating pod on a dead node still does.
Then we force-deleted the old Deployment pod, and only that pod:
$ kubectl -n lab delete pod lhd-6b6b475cf8-k2rfb --force --grace-period=0
pod "lhd-6b6b475cf8-k2rfb" force deleted from lab namespaceThe controller force-detached 361 seconds later, logging maxWaitForUnmountDuration expired, force detaching with duration="6m0s", and the new pod was writing 376 seconds after the delete.
Those six minutes should not have happened with Longhorn. Version 1.13.0 deletes the VolumeAttachment of a pod you force-delete, but only when a pending pod carries an event containing the literal text "Multi-Attach error". Kubernetes stopped writing that text in 1.37.0, 1.36.4 and 1.35.9, so on those releases, going by the code, the cleanup never matches and you get the full timer, which is what we measured on 1.37.2. Longhorn has merged a fix for its next releases, 1.13.1 and 1.12.2 (issue #14182). On Kubernetes 1.36.3 the same delete usually clears within a minute, because Longhorn deletes the attachment and Kubernetes notices on its 60-second sync. We did not run 1.36, but a report on Longhorn's tracker measured a median of 47 seconds to a running pod there and 372 on 1.36.5, close to our 376, and a suspected race, still open (issue #14183), sent 1 of the reporter's 13 force deletes back to the six minutes.
The log line is honest about the risk: "this volume is not safe to detach". The controller cannot tell a node that is off from one that has lost its network and is still writing, and the Kubernetes node shutdown documentation warns that a force-detached volume still in use can be corrupted.
The out-of-service taint skips both waits
On a NotReady node carrying node.kubernetes.io/out-of-service, the pod garbage collector force-deletes the terminating pods and the controller detaches their volumes immediately, without the safety check. Kubernetes says to confirm the node is "already in shutdown or power off state" before adding the taint, and to remove it yourself once the node is back. That condition is the entire risk. Check the hypervisor, the IPMI console or the power strip. Not ping.
Our StatefulSet pods stayed Terminating for 18 seconds after the taint, because the collector runs every 20. The Longhorn one was attached on the surviving node at 26 seconds and writing at 27. Both Longhorn volumes came back from the replica on that node holding every line written up to 3 seconds after we issued the stop, which is as far as the dying VM got.
If that event is on a production cluster right now and you are not sure the node is off, send the paste described at the end of this post through the scoping form before you taint anything.
Volume node affinity conflict: local-path cannot follow
The taint freed the local-path pod as well, and its replacement went straight to Pending:
0/2 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s).Before Kubernetes 1.33 that reason read "had volume node affinity conflict". local-path makes a directory on one node's disk and pins the PersistentVolume to that node by hostname, so there is nothing to attach anywhere else. The data was on a machine with no power. No timer, taint or Longhorn setting moves it. Ours was running 23 seconds after we powered the worker on, 3 seconds after we removed the taint, with all 45 lines intact. A dead disk instead of a dead power supply means a restore, or nothing.
That trade is fine for a cache, or for a database that keeps its own copies on separate nodes, such as a PostgreSQL standby or a three-member MongoDB replica set, which is how a managed database gets built anyway. It is a bad one for a single-replica database on local-path, and k3s marks local-path as the default StorageClass, so that is what a PersistentVolumeClaim with no class gets on a stock install. It is one of the defaults that separate k3s from RKE2 and kubeadm.
Letting Longhorn do it at 02:00
Longhorn ships node-down-pod-deletion-policy set to do-nothing. For the second cut we changed it:
$ kubectl -n longhorn-system patch settings.longhorn.io node-down-pod-deletion-policy \
--type=merge -p '{"value":"delete-both-statefulset-and-deployment-pod"}'Nobody touched the cluster after the power went. Eviction came at 348 seconds. Longhorn deleted both VolumeAttachments within 3 seconds and had force-deleted the stuck pods by 370, and both Longhorn workloads were writing on the surviving node at 411 seconds, 63 seconds after the eviction. Most of that last minute was Kubernetes catching up: its attach-detach controller re-reads attachments on a 60-second sync, and the new attach started on that sync. The local-path pod stayed Terminating, correctly, since Longhorn has no claim on it.
The setting makes the same bet as the taint, placed by software instead of a person: that a node Longhorn sees as down is not still writing. Where a network split can make a live node look dead, the missing piece is fencing, which is a design question rather than a setting.
The order we set a bare-metal cluster up in follows from the runs. Replicated storage under anything stateful that cannot rebuild itself, for the same reason the address in front of it needs a load balancer that survives a dead node. Longhorn's node-down policy switched on, on a cluster where a down node really is down. A runbook line for the rest: confirm power-off at the hypervisor, taint, watch kubectl get volumeattachments until the dead node's rows go, remove the taint when the node is back. Then cut a worker's power in a maintenance window and time it, because a patch release can move one of the numbers above: 1.36.4 only reworded an event, and that put Longhorn's force delete back on the six-minute timer.
One node, or nothing stateful on the cluster, and none of this applies; you do not need us for it. Otherwise paste the FailedAttachVolume or FailedScheduling event lines from kubectl describe pod, the output of kubectl get volumeattachments, kubectl get pvc run in the stuck pod's namespace, the storage driver with its version, and whether the old node is confirmed off into "Anything we should know?" on the Kubernetes scoping form, choosing "Kubernetes on our own hardware", or send the same on WhatsApp at +254 713 403 044, which takes a long paste as a file. You get back which stage each volume is stuck in, the safe step for it, and the StorageClass and setting changes that let the next node failure clear itself. If volumes have to come off local-path onto replicated storage, we handle that move with you. The diagnosis is free and commits you to nothing.