Infrastructure engineering for East Africa's operators, platforms and regulators See the work →
OpenStack Skip-Level Upgrade: What Epoxy to Gazpacho Changes
Cloud 7 min read

OpenStack Skip-Level Upgrade: What Epoxy to Gazpacho Changes

An OpenStack skip-level upgrade from Epoxy to Gazpacho skips Flamingo, not its changes: 18 nova options go unwarned and nova-api moves to threading.

CE
CloudSpinx Engineering
Infrastructure Engineering, Virtualization, Networking, Automation
29 October 2026
openstackopenstack upgradeslurpskip-level upgrade

"The native threading mode is experimental. Do not use it in production without first testing it in pre-production." That is nova's own concurrency guide for 2026.1 Gazpacho, and the same page says that since nova 33.0.0, nova-api, nova-metadata and nova-scheduler run in native threading by default. Take an Epoxy cloud straight to Gazpacho without one particular environment variable set and those three services come back up in the mode the guide warns about.

CloudSpinx builds, runs and upgrades OpenStack private clouds for organizations in Kenya. For this OpenStack skip-level upgrade, 2025.1 Epoxy to 2026.1 Gazpacho, we diffed the published sample configuration of six core services (nova, neutron, cinder, glance, keystone and placement) across Epoxy, Flamingo and Gazpacho, loaded an Epoxy-style nova.conf into nova 33.0.2, and searched the stable/2026.1 branches of Kolla-Ansible and OpenStack-Ansible. We did not upgrade a running cloud for this post, so there is no downtime figure in it.

Can you upgrade OpenStack from Epoxy straight to Gazpacho?

Yes. Both are SLURP releases, and the skip-level release policy supports and tests a move from one SLURP to the next without installing the release in between, which here is 2025.2 Flamingo. Three lines of that policy decide how the upgrade has to run:

  • It does not promise a rolling upgrade, and expects a SLURP-to-SLURP schedule to end up "requiring some downtime during an upgrade because components will be spanning more than two actual releases."
  • Data migrations have to be finished on the release you are leaving, before you move.
  • An upgrade that spans more than one SLURP cycle is "not tested or required", so a cloud on 2024.1 Caracal goes to Epoxy first and then to Gazpacho.

That Caracal cloud has a clock on it. The OpenStack releases page lists Caracal as Unmaintained and gave Epoxy an estimated move to Unmaintained of 2 October 2026. Unmaintained branches produce no releases, and under the stable-branch policy only the latest eligible Unmaintained branch stays open by default, so once Epoxy moves, each project's Caracal branch can go to end of life unless that project's PTL or unmaintained liaison opts in to keep it open. An upgrade a year belongs in the running cost of a private cloud from the first budget. On Red Hat's distribution rather than upstream, Red Hat's lifecycle sets these dates instead.

Why nova-api runs in native threading after the upgrade

The concurrency mode is not a nova.conf option, so no review of nova.conf finds it. nova 33.0.2 reads one environment variable, OS_NOVA_DISABLE_EVENTLET_PATCHING, as each service starts: true asks for native threading, false for eventlet, and unset means the service's built-in default. That default is native threading for nova-api, nova-metadata and nova-scheduler, and eventlet for nova-compute and nova-conductor. We ran nova's mode-selection code both ways. Unset, the threading default logs this:

WARNING nova.monkey_patch: Service is starting with native threading. This is currently experimental. Do not use it in production without first testing it in pre-production.

Neither Kolla-Ansible nor OpenStack-Ansible sets the variable. We searched the stable/2026.1 branches of Kolla, Kolla-Ansible and OpenStack-Ansible's nova role for it and found nothing, and both tools run nova-api under uWSGI from nova.wsgi.osapi_compute, the module whose default is threading. On either tool's defaults, the version upgrade is also the concurrency change.

nova's concurrency guide and its 33.0.0 upgrade note offer two ways to keep those apart: test threading on Flamingo first, or run eventlet explicitly through the upgrade and switch later as its own change. A skip-level upgrade never runs Flamingo, so the second is the one you have. Where a service runs as its own systemd unit, as on an OpenStack-Ansible or package install, it is one Environment= line in that unit, shown here as a drop-in for the scheduler together with the stop timeout (OpenStack-Ansible can render the same through the role's nova_scheduler_init_overrides and its siblings):

# /etc/systemd/system/nova-scheduler.service.d/override.conf
[Service]
Environment=OS_NOVA_DISABLE_EVENTLET_PATCHING=false
TimeoutStopSec=180

The second line is for a different Gazpacho change, and it belongs on every nova unit, not only these three. nova's graceful_shutdown_timeout went from 60 to 180 seconds for all nova services in 33.0.0, and the note asks for a systemd stop timeout of at least 180 seconds so a service is not killed halfway through shutting down. On Kolla-Ansible, OS_NOVA_DISABLE_EVENTLET_PATCHING has to reach each container's environment instead, and the nova role does not set it, so it is an override you add and prove on staging. Kolla-Ansible's own container stop timeout, docker_graceful_timeout, defaults to 60 seconds on stable/2026.1.

Config options a skip-level upgrade drops without a word

Between the Epoxy and Gazpacho sample files, nova loses 20 of its own options, and 9 more are renamed, 8 of them oslo.cache's [cache] options. Eighteen of those 20 were never marked deprecated in Epoxy, not in its sample file and not in the 31.0.0 source. All 18 went in Flamingo, with the eventlet API server and the remote debugger: enabled_apis, enabled_ssl_apis, the listen address, port and worker count for both the compute API and metadata, the two [remote_debug] options, and eight [wsgi] options from ssl_ca_file to client_socket_timeout. Flamingo's release notes say the eventlet API server is gone. They do not name one of the options.

Where the OpenStack options removed between Epoxy and Gazpacho went
Where the OpenStack options removed between Epoxy and Gazpacho wentOption entries in each service sample configuration for Epoxy 2025.1 that are absent from its Gazpacho 2026.1 sample, split by the release they disappeared in. nova lost 20 in Flamingo 2025.2, the release a skip-level upgrade never installs, and 18 of those were never marked deprecated in Epoxy. neutron, cinder, glance, keystone and placement lost most of theirs in Gazpacho itself. Across the six files, 22 of 46 went in Flamingo.removed option entries, per sample filenova20 + 218 of the 20 never marked deprecated in Epoxyneutron0 + 8cinder1 + 9glance-api1 + 2keystone0 + 2placement0 + 1ALL SIX FILES22 gone by Flamingo, 24 in Gazpacho
  • gone by Flamingo 2025.2
  • gone in Gazpacho 2026.1
Skipping Flamingo skips its install, not its removals: 22 of 46 removed option entries were gone by Flamingo, and 20 of those are nova’s.

Counted per sample file, so an option from a shared library such as oslo.log is counted once in every file that carries it. Renamed options are not in the bars, because the new option still reads a value set under its old name.

Each of those is a setting somebody once chose. A raised max_header_line or a longer client_socket_timeout stopped applying when the API moved under uWSGI, which takes its header and socket limits from its own settings (Kolla-Ansible's uWSGI template sets buffer-size and socket-timeout), and the file still reads as if they apply.

We loaded an Epoxy-style nova.conf with seven of the removed options set into nova 33.0.2, logging at debug level. nova said nothing about any of them. The two renamed [cache] options were read under their new names, and each logged a deprecation warning when read. OpenStack-Ansible's nova role on stable/2026.1 still writes enabled_apis into the nova.conf it renders, and nova ignores that line just as quietly.

The tool that does see them is oslo-config-validator, run against the release you are going to. It reads the options from that release's code, so build Gazpacho's nova into a Python virtual environment against the 2026.1 upper constraints first. Against our file:

$ oslo-config-validator --input-file nova.conf --namespace nova.conf --namespace oslo.log ...
ERROR:root:DEFAULT/enabled_apis is not part of the sample config
ERROR:root:DEFAULT/osapi_compute_listen is not part of the sample config
WARNING:root:Deprecated opt cache/memcache_socket_timeout found
ERROR:root:wsgi/client_socket_timeout is not part of the sample config

That is four of its nine findings: all seven removed options came back as ERROR, both renames as WARNING, and it exited 1. It is silent on two things that matter more. max_concurrent_builds = 0 is a valid integer, so it raised nothing, but the number means something else now. Epoxy documented 0 as unlimited. On Gazpacho it means 1,000 concurrent builds on a compute in eventlet mode and 10 in native threading, and nova-compute logs only that 0 is deprecated, not what it now means. And the validator cannot see an environment variable at all.

Which check catches which nova change on Gazpacho
Which check catches which nova change on GazpachoFour kinds of change an Epoxy nova.conf meets on Gazpacho, against three places to catch them: what nova 33.0.2 logs, what oslo-config-validator reports against Gazpacho’s options, and nova’s release notes. A removed option logs nothing and only the validator names it. A value whose meaning changed, max_concurrent_builds = 0, passes the validator in silence, and nova-compute warns only that 0 is deprecated, not what it now means. The threading default sits outside the file, so only the start-up warning and the 33.0.0 note show it.what nova 33.0.2 logsoslo-config-validatornova release notesRemoved optionosapi_compute_listennothingno log lineERRORnames the optionfeature onlyoptions not namedRenamed option[cache] dead_timeoutWARNINGvalue still appliedWARNINGnames the old oneelsewherein oslo.cache's notesSame value, new meaningmax_concurrent_builds = 0WARNINGcompute: 0 deprecatednothinga valid integer33.0.0 notewhat 0 means nowMode outside the filean environment variableWARNINGexperimental modecannot see itnot in nova.conf33.0.0 notethreading by default
  • caught
  • partly
  • missed
No one check covers all four. The validator is the only one that names a removed option, and it says nothing about the other two kinds.

The validator checks the file against the option definitions it is given, from installed code or from a generated YAML file, which is why they have to come from the release you are moving to.

If you would rather have the validator run against your own files by someone who has read both releases' notes, tell us about the cloud through the OpenStack scoping form and send the files, passwords taken out, on WhatsApp or to [email protected].

Rack servers with hot-swap SAS drive bays and green activity lights, the kind of compute node an OpenStack upgrade reaches last

The OpenStack upgrade order we would run

Read Flamingo's notes as well as Gazpacho's: nova alone has 7 upgrade notes in 32.0.0 and 12 in 33.0.0, and skipping a release skips installing it, not its changes. Some of those changes are loud rather than hidden. The nova-api-wsgi and nova-metadata-wsgi scripts are gone in 33.0.0, and keystone removed keystone-wsgi-public and keystone-wsgi-admin in Flamingo, so a web server config that still points at one of those script paths has nothing to load, and that shows on the first request rather than weeks later.

The rest is order. Rehearse on a staging cloud built from a dump of production's databases. On Epoxy, before anything moves, run nova-manage db online_data_migrations until nothing is left, then the validator on every service's file against Gazpacho's options, then pin the three nova services to eventlet and raise every nova service's stop timeout to at least 180 seconds. Upgrade the deployment tool next. For a skip-level upgrade, Kolla-Ansible's guide asks for ansible-core at a version the target release supports. Then take the API window the policy warns about: stop every nova service except the computes, run nova-status upgrade check on the new code, and start nova-conductor first and nova-api last, which is the order nova's upgrade guide gives. Computes follow. If Ceph sits under the cloud and runs Squid, its end of life is a separate hop, and the Proxmox version of that upgrade shows how many steps it hides; keep it out of the same window. Native threading comes weeks later, one service at a time, once a staging cloud has carried real load on it.

A Kolla-Ansible cloud on Epoxy with a staging copy that mirrors production, run by a team that has carried its own upgrades before, does not need us. Pin the mode, run the validator, follow the guide. And if the cloud serves one team with no tenants of its own, ask whether it should be OpenStack at all before paying for a skip-level upgrade.

A cloud still on Caracal, built by an integrator who has since left, with config files nobody has diffed in two years, takes two hops of these changes with nobody left who knows which settings were deliberate. Tell us the deployment tool and its version and how many controllers and computes you run through the OpenStack scoping form, and send every service's config file with the secrets stripped on WhatsApp at +254 713 403 044 or to [email protected]. You get back each option marked removed, renamed or changed in meaning, the mode every nova service will start in, the hops your cloud needs, and one figure for running the upgrade with your team, rehearsal first. There is no charge for that answer and you are not committed to anything by asking.

WhatsApp