Kubernetes External IP Pending: LoadBalancer on Bare Metal

kubectl get svc prints <pending> under EXTERNAL-IP until a load balancer controller in the cluster gives a Service of type LoadBalancer an address. In a cloud the provider's controller does it. On your own servers with MetalLB in layer 2 mode, the fix is three pieces: MetalLB itself, a pool of free addresses on the nodes' own subnet, and an announcement of that pool to the network. Stop before any one of them and the Service stays broken, and in our runs none of those three stops put a warning on the Service itself.

We ran the k3s case on k3s v1.37.2 and the rest on a kind cluster (Kubernetes v1.37.0 on nodes kind builds with kubeadm), adding MetalLB v0.16.2 one object at a time, then killed nodes under a request loop to see what layer 2 failover costs once the address works. The only failover measurement we found was a MetalLB issue filed against v0.7.2, none of the results we read split the cost between MetalLB and kube-proxy, and most of the failed requests we counted turned out to be kube-proxy's. CloudSpinx builds and runs on-premises Kubernetes in Kenya, on bare metal or on a Proxmox or OpenStack platform when the data has to stay in the country, and the load balancer is one of the pieces a cloud hands you that a cluster on your own metal has to have built.

Which controller should fill in EXTERNAL-IP?

Check the distribution before you touch any YAML, because <pending> means something different on each.

kubectl get pods -A | grep -i -E 'svclb|metallb' finds ServiceLB or MetalLB if either is running, and no other implementation. On a kubeadm cluster nobody has added a load balancer to, it prints nothing, and nothing is answering.

kubeadm and kind: pending with no events at all

Two minutes after kubectl expose deployment web --type=LoadBalancer --port=80 on kind, with nginx behind the Service:

$ kubectl get svc web
NAME   TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
web    LoadBalancer   -            <pending>     80:31161/TCP   2m
$ kubectl describe svc web | sed -n '/Events/,$p'
Events:                   <none>

Nothing is wrong as far as the cluster is concerned. The Service exists, its NodePort is allocated, and no controller has claimed it. RKE2 should land in the same place unless ServiceLB was switched on, going by its docs (we have not put a LoadBalancer Service on RKE2), and k3s's own documentation states the upstream position plainly: Services of type LoadBalancer "will remain pending until one is installed".

k3s: pending because Traefik already holds port 80

Stock k3s ships Traefik with its own LoadBalancer Service on ports 80 and 443, and ServiceLB satisfies it by running a pod with those host ports on every node. Expose your own app on port 80 and it waits for a port that is already taken. Sixty seconds after the same expose, the reason sat on a pod in kube-system, not on the Service:

$ kubectl -n kube-system describe pod svclb-web-a1031b03-vqkwn | sed -n '/Events:/,$p'
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  60s   default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.

kubectl describe svc web showed two Normal events, "Ensuring load balancer" and "Applied LoadBalancer DaemonSet", which read like success. The same Deployment exposed on 8080 got the node's address within a second and answered.

So on k3s, move the port, or send HTTP through Traefik with an Ingress so one address serves every hostname. If you want MetalLB instead, the k3s docs say to start every server with --disable=servicelb, and warn that Traefik's own Service then stays pending too until the new load balancer is in.

If you are changing ingress controllers at the same time, what broke when we moved Ingress objects to Gateway API has its own post.

MetalLB installed and the address still does not answer

Installing MetalLB from its v0.16.2 manifest brought up a controller and three speakers. The Service stayed at <pending> with no events. The only trace we found was one line in the controller's log, "failed to handle service, no retry", which never mentions a pool. MetalLB's troubleshooting page is right that the controller is the component to check, but you have to know to look.

An IPAddressPool fixed the pending in under a second, and left a state that looks fixed and is not:

$ kubectl get svc web
NAME   TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)        AGE
web    LoadBalancer   -            198.51.100.200   80:31161/TCP   3m52s
$ curl -s -m 3 -o /dev/null -w '%{http_code} %{time_total}s\n' http://198.51.100.200/
000 3.001955s
$ ip neigh show 198.51.100.200
198.51.100.200 dev br-53da00f1a772 INCOMPLETE

An address in EXTERNAL-IP that nothing answers, still failing 20 seconds later, with one Normal event (IPAllocated) on the Service. Nobody is replying to ARP for it, because a pool only lets MetalLB assign addresses. MetalLB's configuration page puts it in one sentence: an L2Advertisement "must be associated to the IPAddressPool" before the address is advertised. The tell is the event that is missing, nodeAssigned. We applied an L2Advertisement and the first request answered 0.85 seconds later, with the event naming the node that took the address:

Normal  nodeAssigned  3s  metallb-speaker  announcing from node "lb-worker" with protocol "layer2"

The node it named matters. kubeadm labels every control-plane node node.kubernetes.io/exclude-from-external-load-balancers, and MetalLB will not announce from a node with that label. In our runs the control plane's speaker never announced. Put those two rules together and a kubeadm cluster made only of control-plane nodes has nowhere to announce from. We did not run that case, and MetalLB's troubleshooting page gives the speakers' --ignore-exclude-lb flag as the way round it. If kube-proxy runs in IPVS mode, MetalLB's installation page also requires strictARP: true; ours ran iptables mode, kind's default, and needed nothing.

What a dead node costs behind MetalLB layer 2

With the address working, we sent requests one at a time with a 0.1 second pause between them, each a new TCP connection with a one second timeout, from one Linux host on the same bridge as the nodes, and killed nodes. The nodes are containers on one VM, so a kill takes the node's network away at once, which stands in for pulling the power.

Killing the node that announced the address cost 6.4 seconds of address move. Every request failed from 0.65 to 6.25 seconds, the host's ARP entry flipped to the surviving worker at 6.4 seconds, and the first answer came back at 7.4. That is under the roughly 10 seconds above which MetalLB's layer 2 page asks for a bug report, and the speaker's log shows memberlist suspecting the dead node within about four seconds.

Then 38 of the next 71 requests failed anyway. The surviving node's kube-proxy kept sending new connections to the pod on the dead node until that node went NotReady at 51.3 seconds and its endpoint was marked not ready at 52.4. That is the 50 second node grace period, the window the node failure post measured when a worker lost power, showing through the load balancer. Killing the other worker, the one not announcing, showed it plainly: the address never moved, and 37 of 84 requests still failed over the 46.5 seconds before its endpoint went.

Nearly every failure here sat out our full one second timeout. A browser or a mobile app with a longer connect timeout would hang for longer on each one. If you would rather know your own cluster's number before a node dies, that is a failover test we can scope with you.

Two loose ends from the same run. Bringing the killed node back moved the address home 47 seconds after the restart and cost our client no request. And three requests failed between 81.8 and 84.3 seconds, after the dead pod's endpoint was marked not ready, which we did not trace.

externalTrafficPolicy Local took the 45 seconds away

We patched the Service to externalTrafficPolicy: Local and repeated both kills. Killing the announcing node failed every request from 0.4 seconds until the address moved at 7.5 seconds, 10 in all, and every one of the 533 requests after that answered. Killing the other node cost nothing: 341 of 341 answered over 40 seconds. With Local, the node holding the address sends traffic only to pods on itself, so a pod on some other dead node is never in the path.

It is not free. In layer 2 mode every request for an address enters through one node, and MetalLB's usage page is blunt about Local: pods that are not on that node "receive no traffic". The replicas elsewhere are standby, not capacity, so the pods on any one node have to carry the whole load. If they cannot, keep Cluster and live with that window, or, if your routers speak BGP, look at BGP mode, where the same page says the routers spread Local traffic across every node running one of the pods. We did not run BGP. For a customer-facing API we would take Local and size each node's pods for all of it.

The config we start a bare-metal cluster with

A pool on free addresses in the nodes' own subnet, outside the DHCP scope, then an advertisement covering it:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lan-pool
  namespace: metallb-system
spec:
  addresses:
  - 198.51.100.200-198.51.100.210
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lan-l2
  namespace: metallb-system

Then externalTrafficPolicy: Local on the Services behind it, and the test that tells you whether any of it holds: kill the announcing node under load and count the failures.

A single k3s box does not need this. ServiceLB gives every Service the node's address as long as the ports do not collide, and a cluster in EKS, AKS or GKE gets its load balancer from the provider. Where it pays is several nodes on your own metal with something behind the address that has to keep answering when one of them dies. The distribution under it matters too, and k3s, RKE2 and kubeadm differ in more than the load balancer.

Paste the output of kubectl get svc -A | grep -E 'NAME|LoadBalancer', the distribution and its version, and how many nodes sit on which subnet into the "Anything we should know?" box on the Kubernetes scoping form, choosing "Kubernetes on our own hardware", or send it on WhatsApp at +254 713 403 044. You get back why each pending Service is pending, the pool and advertisement for your network, and what a node failure is likely to cost the apps behind the address. If we run the cluster afterwards, the load balancing, monitoring and upgrades come as one monthly figure. The diagnosis is free and commits you to nothing.