etcd Quorum Lost: Kubernetes Does Not Go Read-Only

Two guides on etcd quorum loss say Kubernetes goes read-only. One, written for Talos, puts it in a sentence: with two of three nodes down, "the remaining single node cannot form quorum and the cluster becomes read-only." We powered off two of three k3s servers, and the survivor refused reads as well as writes. The first kubectl get after the cut timed out while k3s was still up, so that much is the API server's own behavior. k3s itself exited 16 seconds after the last write landed, and what kept serving was whatever had been running before, minus every part that needed the API to stay correct.

The cluster was three Ubuntu 24.04 VMs on Proxmox running k3s v1.37.2 with its embedded etcd, which reports version 3.7.1. Each server also ran one replica of a small web Deployment behind a NodePort, because k3s servers take workloads unless you taint them, so losing two servers here also lost two of the three pods. Once a second we read the API, wrote to it, asked etcd for its status and sent a request to the app from outside the cluster. Bringing a control plane back from this is part of what managed Kubernetes from CloudSpinx covers for teams in Kenya, so we also timed both recovery routes k3s documents.

What still works when etcd loses quorum?

The containers that were already running, and the forwarding rules already written into the kernel. Nothing else.

The pod on the surviving server kept answering through k3s's exit and the restart after it, and so did the NodePort in front of it, on rules nothing could update: k3s exited, and the copy systemd restarted never got anything but errors from its own API. What could not change was the list of pods behind the Service. Two of the three sat on powered-off servers, taking them out of rotation is a write to the API, and so the survivor kept sending them a share of the traffic for as long as quorum stayed lost. A healthy control plane stops after about 50 seconds.

Then we stopped the app container on the survivor with crictl stop, standing in for a crash. Nothing restarted it. k3s runs the kubelet inside the same process as the API server, and while k3s sat waiting for etcd the kubelet never came back: k3s logged "Running kubelet" within three seconds of the restart, but the kubelet logged nothing of its own and its port was still closed when we looked, about three minutes in. The container stayed down until we recovered etcd. Kubernetes' own etcd operations page says that with a majority of members gone "the scheduled pods might continue to run". On our k3s survivor, the one that stopped stayed stopped. On kubeadm the kubelet is its own systemd service, which is one of the differences between k3s, RKE2 and kubeadm, and we did not run kubeadm.

Why k3s exits 11 to 16 seconds after etcd loses quorum

The readiness check failed on the first probe after the cut, naming etcd. Eleven seconds after the last write the scheduler failed to renew its lease, five seconds after that it gave the lease up, and the whole process went with it:

Error from server (InternalError): ... [-]etcd failed: reason withheld
"Failed to renew lease" lock="kube-system/kube-scheduler" err="context deadline exceeded"
E1110 10:39:33.855304    4340 server.go:335] "Leaderelection lost"
k3s.service: Main process exited, code=exited, status=1/FAILURE

The scheduler holds a leader lease that it must renew within 10 seconds, the default renew deadline, and renewing it is a write. When the write cannot commit, kube-scheduler logs that line and exits the process. In kubeadm that process is the scheduler alone. In k3s it is everything: the API server, the controllers and the kubelet. Whichever component gives up its lease first takes it down. That was the scheduler here, and the cloud controller manager and the controller manager in our two later quorum losses, 11.5 and 12.0 seconds after the last write.

systemd restarted k3s 5 seconds later, and it sat in activating for as long as quorum stayed lost, its etcd logging failed to publish local member to cluster through raft every 15 seconds. From then on the API answered with the first two lines below, and etcd's health check with the third:

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods)
error: failed to create deployment: apiserver not ready
https://127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded

Asking etcd directly did not help either. Its FAQ says serialized reads can be served by any member, but the restarted survivor's etcd never opened for clients at all: a member serves nobody until it has published itself to the cluster through raft, which takes quorum, so etcdctl get --consistency=s could not even connect. A MariaDB Galera node without quorum behaved the same way in our OpenStack controller lab: no read-only mode, just refusal.

One of three is a pause, two of two is an outage

With one of three servers off, quorum holds. We powered off the etcd leader on purpose. One write hung for its 5-second timeout and the next went through 7.2 seconds after the last good one, because the survivors took 6 to 7 seconds to elect a new leader. k3s sets etcd's election timeout to 5,000 ms, five times etcd's 1,000 ms default, the time a follower goes without hearing from a leader before it stands for election itself.

A second control-plane server does not buy even that. The etcd FAQ's failure tolerance table gives a two-member cluster a tolerance of zero, the same as one member, because a majority of two is two. We ran it: with two members, powering one off left the survivor's API refusing requests the same way losing two of three had, and k3s exited 11.5 seconds after the last write. Either server failing now takes the API down, so the second server doubles the machines whose loss stops it.

A reset or a restore passes through that state. Either leaves you one member, each server you rejoin adds one, and the cluster sits at two with no tolerance until the third arrives. Our two-member run was exactly that, halfway back from a reset.

When the missing member came back, nothing needed doing. The survivor took its first write 22.6 seconds after we powered the other server on, boot included, and passed its readiness check at 24.9. The etcd FAQ describes the same thing for power loss: a member "replays the log to the point of failure and resumes cluster participation".

How do you recover etcd quorum on k3s?

Wait, if the lost servers can come back. If they cannot, reset from a survivor whose disk is intact, and restore a snapshot only when none is, because a restore throws away everything written after the snapshot.

If a cluster of yours is sitting in one of those rows now, send us the output listed at the end of this post through the scoping form before you pick one.

The reset is three steps: systemctl stop k3s, k3s server --cluster-reset, start k3s again. The catch is that the restored state includes the dead servers. All three nodes read Ready and all three pods stayed in the Service, so the survivor kept routing to pods on servers that were off until the node controller marked those nodes NotReady about 50 seconds after it started, the same grace period we timed in the node failure post. Every request answered from 85 seconds after the restart. Our survivor had been the etcd leader. A follower may not hold the last writes the other two committed before they died, and we did not test that case.

The servers you bring back must forget their old data. The message the reset prints says to back up and delete it, and k3s's restore steps say to remove /var/lib/rancher/k3s/server/db/ on each of the other servers before starting it. Ours came up with its old data, logged this and waited:

This node has been removed from the cluster - please restart k3s to rejoin the cluster

On v1.37.2, one systemctl restart k3s later it found a tombstone file, left the old database in an etcd-backup-<timestamp> directory and was back in the member list when the restart returned, 13 seconds on. We did not delete anything by hand.

Restoring a snapshot is the same command with --cluster-reset-restore-path=<file>. We took one by hand and kept changing the cluster for about a minute: 60 writes from our probe loop, a ConfigMap, and a Deployment pinned to the survivor with its pod running. After the restore the ConfigMap and the Deployment did not exist, and with no pod for it in the restored API, the kubelet removed the running container about 4 seconds after the API first passed its readiness check. On k3s defaults the newest scheduled snapshot can be up to 12 hours old (the schedule is 0 */12 * * *, five kept per server), and the snapshots sit on each server's own disk unless you configure S3. Copy /var/lib/rancher/k3s/server/token off with them: k3s's docs say a restore with a different token leaves the snapshot unusable.

Where read-only is still right

About the workloads, in a narrow way. What the cluster knows stays exactly as it was when quorum went, because nothing can write to it, and that includes the changes you need. The Service kept its dead pods for the whole outage, and the stopped container stayed stopped. Plural's etcd explainer says the control plane becomes "read-only and then unresponsive", which is nearer what we saw, except that our reads failed from the first probe.

On clusters we build, three control-plane servers in three failure domains is the floor, and that redundancy is what our 99.99% HA uptime SLA is signed against. A single-server cluster has no quorum question: if the server dies you restore from a copy, which on k3s's default SQLite datastore is the server/db directory plus the token, and you do not need us for it.

For anything bigger, write "etcd quorum lost" in "What do you need?" on the Kubernetes scoping form, choose "Kubernetes on our own hardware", and "Urgent, something is broken" if it is. Paste into "Anything we should know?" what each server still up prints for kubectl get --raw='/readyz?verbose', for etcdctl endpoint status --cluster -w table if you have etcdctl (k3s does not bundle it), and for k3s etcd-snapshot list or rke2 etcd-snapshot list (or ls -l of the snapshots directory if that fails), errors included. Then add the distribution and version, how many control-plane servers are down and whether their disks survived. That box takes 5,000 characters; journal excerpts and anything longer go on WhatsApp at +254 713 403 044 as a file. You get back which route fits (wait, reset from a survivor, or restore), the steps in order, and what each will lose. The diagnosis is free and commits you to nothing.