If kubectl just told you a Kubernetes certificate expired, in the words x509: certificate has expired or is not yet valid, your applications are probably still up. When we let a kubeadm cluster's certificates run out, the control plane stopped taking instructions and the pods carried on serving. The outage came at the next reboot, and it outlived the fix most answers give: after kubeadm certs renew all, kubectl worked, the node read Ready, and the application still did not answer.
CloudSpinx runs Kubernetes for organizations in Kenya through our managed Kubernetes service, and this one belongs to the self-managed kind: kubeadm clusters on servers in Nairobi or in a client's own rack, where nobody else renews anything. So we let one expire on purpose. kubeadm v1.36.5 on a single Proxmox VM, with kubeadm's own certificateValidityPeriod field set to 25 minutes so that a year passed inside an hour, an nginx Service, and a probe every 15 seconds. The pod network is flannel and the one node is untainted so pods can run on it. Nothing else differs from stock, and every time below comes from that run.
What expires, and when
Eleven certificates, in the same minute. kubeadm init gives them one year, the 8760h default that kubeadm config print init-defaults prints, and gives its three certificate authorities ten. kubeadm certs check-expiration lists them: the API server's serving certificate, its client certificates for etcd and for the kubelets, etcd's own three, the front proxy client, and the client certificates inside admin.conf, super-admin.conf, controller-manager.conf and scheduler.conf.
Nothing renews them on a timer. kubeadm renews all of them during a control-plane upgrade, and the kubeadm certificate guide says that covers you if you upgrade "less than 1 year in between each upgrade". The cluster that expires is the one nobody upgraded for a year, which is also the cluster falling out of patch support.
The kubelet is the exception, and it matters later. It asks the control plane for a new client certificate at 70 to 90 percent of the old one's life, a jittered threshold in the kubelet's rotation code. Ours rotated at 82.6 percent, five minutes before everything else expired, and got a certificate valid for a full year.
x509: certificate has expired or is not yet valid
kubectl printed this, and the controller manager and the scheduler logged the same x509 error:
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-10-21T04:21:32Z is after 2026-10-21T04:20:26ZRead the end of it. is after followed by a date means a certificate expired at that moment. If it says is before, the certificate is not valid yet, which almost always means a clock is wrong, on this machine or on the one that signed it.
Adding --insecure-skip-tls-verify only changes the message, to error: You must be logged in to the server (Unauthorized). The client certificate inside admin.conf expired in the same minute, and the API server checks a client certificate's dates on every request, not once per connection.
What kept running after the Kubernetes certificates expired
- kept working
- half working
- failed
We probed every 15 seconds for those 1 m 47 s and then rebooted. How long the open connections would have lasted on their own, we did not measure.
The controller manager and the scheduler went first. Ten seconds after the expiry both failed to renew their leader-election leases and exited, and the restarted copies logged the same x509 error on every request. The API server could no longer open new connections to etcd, whose serving certificate had expired too. It kept the connections it already had, and so did the kubelet, which posted a node heartbeat 65 seconds after the expiry and renewed its lease at 86.
The workload never noticed. The NodePort, the ClusterIP and cluster DNS answered on every probe until we rebooted, 1 minute 47 seconds after the expiry. We stopped an nginx container by hand and the kubelet had it running again about a second later, and reported the restart to the API server.
So a cluster can sit in this state looking healthy to its users while nothing can be deployed, scaled or rescheduled. We did not watch long enough to say how long those open connections last, which is the reason to treat a quiet expired cluster as one restart away from an outage.
The reboot takes the cluster down
We rebooted the node. It was back on SSH 24 seconds later, and the application never came back with it: the NodePort, the ClusterIP and DNS failed from the first probe. The API server crash-looped, timing out against etcd with error creating storage factory: context deadline exceeded, because every new connection between them now failed the certificate check. The kubelet could not register the node. nginx, CoreDNS, kube-proxy and flannel all stayed Exited, because the kubelet learns about those pods from the API server. Only the four control-plane static pods start from files on disk.
On hardware you own, nobody schedules that reboot for the certificate. It is a kernel update, a host migration or a power cut, and it lands on a cluster that looked fine that morning.
How to renew kubeadm certificates
The renewal itself is quick. On each control-plane node:
sudo kubeadm certs check-expirationto confirm the dates. It works with the API server down.sudo kubeadm certs renew all, which took about two seconds.- Restart all four control-plane components: kube-apiserver, kube-controller-manager, kube-scheduler and etcd. kubeadm's own last line tells you to.
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/configandsudo chown $(id -u):$(id -g) $HOME/.kube/config, or your ownkubectlstays on the expired certificate.
Two things in that list did not go the way we expected.
The renewal ignored our 25-minute setting. With the API server down it could not read the kubeadm-config ConfigMap, printed Falling back to default configuration, and issued certificates for a year. If you run a custom certificateValidityPeriod, check the new dates afterwards.
After the reboot, the documented restart stopped nothing. The guide says to move the static pod manifests out of /etc/kubernetes/manifests, wait 20 seconds and move them back. We left them out for 93 seconds and the kubelet stopped none of the three control-plane containers still running; the API server was already in crash-loop back-off. On our reading of the kubelet source, it skips pod deletions until it has heard from every pod source, the API server included, and after a reboot with dead certificates it has not. Stopping the containers directly worked, and here the kubelet restarted each stopped container at once:
sudo crictl stop $(sudo crictl ps -q --name kube-controller-manager)
sudo crictl stop $(sudo crictl ps -q --name kube-scheduler)Our API server came back on its own, restarted by the kubelet the same second the manifests went back, and etcd accepted connections on its renewed certificate without a restart. The controller manager and the scheduler did neither, and the controller manager is what kept the application down.
Running, Ready, and still down
- working
- down
The stretch from 5:46 to 14:40 is how long we watched before acting, not a timer. Nothing in the cluster changed during it.
kubectl get nodes answered 4 minutes 49 seconds after the reboot and said Ready. By 5 minutes 46 seconds every pod was running again, and kubectl get pods -A showed 1/1 Running for all of them except the scheduler at 0/1. The application still did not answer, and neither did cluster DNS.
The controller manager and the scheduler were still the processes that had started at boot, before the renewal, presenting the expired client certificates they had read from their kubeconfig files. Every request came back Unauthorized. With no controller manager, nothing updated the EndpointSlices, so the web Service still pointed at pod IPs from before the reboot, and cluster DNS, whose pods had also come back on new addresses, stayed silent with it. Nothing changed by itself for almost nine minutes, until we stopped those two containers. The kubelet restarted both at once, the controller manager took its leader lease 16 seconds later, and the NodePort answered at 19.
So confirm a renewal by calling a Service, or by comparing kubectl get endpointslices -A with the pod IPs in kubectl get pods -A -o wide. Never by kubectl get nodes.
If your cluster is in this state right now, send the error line and kubectl get pods -n kube-system on WhatsApp at +254 713 403 044 and we will tell you which component is still on the old certificate.
Put the date where someone will look
The certificate guide's answer is to upgrade at least once a year, and it is the right one, because the upgrade renews every certificate and you need it to stay on a release upstream still patches anyway. Between upgrades, run kubeadm certs check-expiration from a scheduled job that alerts a person, and know that the API server already exports apiserver_client_certificate_expiration_seconds, a histogram of the remaining lifetime on the client certificates that authenticate its requests. On k3s the date mostly takes care of itself: it renews client and server certificates within 120 days of expiry every time it starts (90 days on releases before May 2025), so only a server that has not restarted in that window gets here.
If check-expiration shows months left and you upgrade every year, you do not need anyone for this. If the dates are close, or your cluster is already in one of the states above, send us the kubeadm certs check-expiration output from each control-plane node and your kubectl version through the Kubernetes scoping form. We reply with the date your cluster stops taking instructions and whether the answer is a renewal now or the upgrade it is overdue for, free and with nothing to sign. Certificate rotation is part of what our managed Kubernetes operations cover, and on clusters we build with redundancy we sign a 99.99% HA uptime SLA. It is also a fair question for anyone you are hiring to run Kubernetes: ask where they keep the certificate dates.