Two controllers looks like the sensible middle for a first OpenStack build. It needs a third less control-plane hardware than three, every service on it has a twin, and if one server dies the other should carry the cloud. On a hardware quote it reads as redundancy.
It is not, and the reason is two services that vote. We ran both in a lab: MariaDB with Galera replication, which holds the cloud's state, with the Galera timeouts Kolla-Ansible's template leaves at their defaults, and RabbitMQ 4.0 with quorum queues, which Kolla now declares by default. On two Galera nodes, killing one left the survivor refusing every write within 6 seconds, and a read we sent later failed too. On two RabbitMQ nodes the queue stopped answering even when the other node was shut down cleanly, the way you would for patching. On three of each, crashing a node the client was not using cost one database write a 6-second wait and nothing more.
How many controller nodes does OpenStack need?
Three, or five if the cloud has to survive two controllers failing at once. Never two. Kolla-Ansible's production architecture guide says so in plain words: deploying two instances of the database and the queue "do not provide (by default) any HA as a failure of one causes a failure of the other one", so the recommended number is three, "where 1 node failure is acceptable". One controller is a lab.
The arithmetic is short. Both services keep working only while more than half their members can see each other. Two out of three is more than half. One out of two is exactly half, and half is not a majority, so the survivor of a crashed pair has to stop. RabbitMQ's quorum queue documentation prints it as a table: two nodes tolerate zero failures, three tolerate one.
One dead controller out of two
Each node in our lab was a process on one machine: MariaDB 11.4.9 with Galera 4.25, and RabbitMQ 4.0.5 on Erlang 27. A client wrote to the first database node about twice a second while we sent kill -9 to another, which is as close as one machine gets to pulling a server's power cord.
On the pair, the write in flight when its partner died hung for 5.9 seconds and came back with an error. Every write after it failed instantly, 56 failures out of 80 writes in the run, and so did a read. The in-flight write returned the first line below, the later calls the second, and the node's status showed why:
ERROR 1180 (HY000) at line 1: Error while appending streaming replication fragment(provider status: Not connected to Primary Component)
ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
wsrep_cluster_size=1 wsrep_cluster_status=non-Primary wsrep_ready=OFFKolla's guide says the database and the queue together "provide the central, persistent storage of information about the cloud", and Nova's own notes place the broker between any two Nova components. We did not run Nova or Neutron on top. The lab stops at the two services they cannot run without.
ERROR 1047: WSREP has not yet prepared node for application use
Here that error means the node has dropped out of the Primary Component, and refusing queries is how Galera stops two halves from taking different writes. On a two-node cluster a crashed partner is enough to cause it. Bring the dead node back and the pair re-forms: ours was Primary again within 1.3 seconds of the killed node starting, with nobody running a bootstrap, and all 24 writes that had succeeded before the crash were still there. If the dead node is not coming back, MariaDB's quorum documentation gives the manual step, SET GLOBAL wsrep_provider_options='pc.bootstrap=true'; on one survivor, and warns never to run it on both sides of a split, which leaves two clusters taking different writes.
A clean shutdown, and why it saves only the database
Shut the second node down properly and Galera carries on: 74 writes out of 74, none slower than 0.35 seconds. MariaDB's weighted-quorum rule counts the majority against the last Primary Component "minus any nodes that left gracefully", so after a planned stop the node that is left holds all the vote that still counts.
RabbitMQ gives no such pass, and it says so before you act:
$ rabbitmq-diagnostics -n r2@localhost check_if_node_is_quorum_critical
Checking if node r2@localhost is critical for quorum of any queues/streams ...
queue 'qq2' in vhost '/' will become unavailable if node r2@localhost stopsIt did. After the clean shutdown, two publishes and a read on the surviving node hung for our client's full 20-second limit, and a third publish for a full 60. On a two-controller Kolla cloud, then, patching one controller stalls the quorum queues that Nova's services talk through.
Three controllers, crashed one at a time
The same kill -9 against one of three Galera nodes held the write in flight for 6.03 seconds, and all 81 writes in the run succeeded. The pause landed between 5.9 and 6.2 seconds in all five Galera crashes we ran, each against a node the client was not writing to. Kolla's ProxySQL sends writes to one node at a time, and losing that one is a test we did not run. We did not trace which timer sets it; Galera's default suspect timeout is 5 seconds, and Kolla's template changes none of Galera's timeouts. On three RabbitMQ nodes, the next publish two seconds after the crash answered in 23 milliseconds.
Three buys exactly one failure. Killing a second database node 10 seconds after the first produced the two-node result again, ERROR 1047 on every call, and a second RabbitMQ crash hung the queue the same way. Five controllers buy two failures, and Kolla's guide says higher numbers usually provide no benefit. A Proxmox cluster runs on the same majority arithmetic, with Corosync doing the counting.
One catch for anyone planning to start with two and add the third later. In our run, a queue created while RabbitMQ had two nodes still had two members after the third node joined, so losing either of the two nodes it lived on would still have stopped it. RabbitMQ grows existing queues when told to, with rabbitmq-queues grow, or when continuous membership reconciliation is on, and its documentation lists that setting as off by default. Buy the third controller at the start.
How many servers an OpenStack cloud needs in total
The controllers are the fixed part of any OpenStack private cloud design. Ceph, if the cloud keeps its volumes there, runs its own vote: its monitor reference recommends at least three monitors for a production cluster. Compute is sized by the workload, and it wants one host more than the instances need, because the instances on a dead host can only restart where there is room for them. Kolla's deployment host is just the machine you run kolla-ansible from, and its sample inventories use localhost.
So the smallest layout that survives a server failure is three servers, each carrying a controller, Ceph and compute. It saves hardware, and the database then shares memory with tenant VMs, which is the trade to watch as the cloud grows. The roomier shape is three controllers plus compute and storage that grow on their own, and what private cloud costs in Kenya shows what three servers that never run your workload do to the budget.
Where two controllers still earn their place
As a copy of the data. Galera is synchronous: MariaDB's architecture page says that when a transaction commits, every node in the Primary Component is "guaranteed to reach a consistent state". So the survivor of a stalled pair holds every committed write, and if its partner is gone for good the database comes back by bootstrapping the survivor by hand, not by restoring last night's backup. That is worth having. It is not availability, and RabbitMQ is blunter about its half: a quorum queue that loses its majority for good has to be force deleted and recreated.
For the database alone the belief can be made true cheaply. Two Galera nodes plus the arbitrator, garbd, kept writing through a crash, 59 writes out of 59, and because the arbitrator votes without storing data it can run on a small third box. RabbitMQ's documentation says quorum queues need a minimum of three cluster nodes to provide fault tolerance, though, so the third box ends up running the queue too, and at that point it is a third controller.
With no AWS, Azure or Google region in Kenya, a private cloud in the country is one way to keep data here, and its control plane has no provider to fall back on.
If one team runs fewer than about a hundred VMs with no self-service, skip OpenStack: Proxmox is the better fit. Otherwise, send us the server list you run or plan to buy: how many hosts, which roles each carries, the RAM in each, and the deploy tool and its version. Paste it into the notes box of the OpenStack scoping form or send it on WhatsApp to +254 713 403 044. You get every role marked enough, short or exposed, what one failed server takes down in that layout, and the change we would make. The 99.99% HA uptime SLA on our OpenStack work is written for redundancy we build, and by this test two controllers are not redundancy. The review is free and binds you to nothing.