Surviving the uncharted: when dedicated OpenStack expertise is your best ally in disaster recovery 

Some support cases are routine. Others take you off the documented path entirely, into territory where the only way forward is deep, hands on open source expertise. This series looks at how Canonical Support navigates the unexpected: cases where standard playbooks aren’t enough, and a support engineer helps a customer find a solution in real time.

This is one of those cases. A customer’s production OpenStack environment went, overnight, from fully operational to unable to provision, scale, or manage a single resource.

What happened

OpenStack is not a single piece of software. It is a coordinated architecture of services that manage core infrastructure components like compute, networking, identity, storage, all of which depend on a shared database that keeps a record of the current state of the cloud: what is running, where, and how it is configured.

During routine operations, an action targeted the database cluster underpinning that control plane. Within seconds, the layer that lets operators manage the cloud, provision, migrate, scale, was gone.

The support case opened the same day.

The first problem: the backup

The team’s first question was straightforward: when was the last backup taken, and had it ever been tested as a restore? The available backup was old enough that it no longer reflected the cloud’s current state. Virtual machines had moved, resources had been reallocated, credentials had rotated since it was captured.

Restoring it directly risked producing a control plane that described a cloud that no longer existed. That ruled out the simplest path, and left something harder: rebuild the control plane around live, running workloads, without breaking them.

This is worth pausing on, because it is a common gap even in mature operations. A backup schedule tells you data is being captured. It does not tell you the restore will work, or that the restored state will still match reality by the time you need it. The two need to be validated together, on a cadence tied to how quickly the environment actually changes.

The thing that didn’t break

Before touching anything, the assigned support engineer checked what was still running, and found that nothing had actually gone down. No workloads were lost. The business kept running. What had been lost was the ability to manage the cloud: no new provisioning, no migration, no scaling.

This is a property of OpenStack’s architecture worth understanding on its own terms. The control plane manages the state of the cloud; the data plane does the actual work, compute, networking, storage. These layers are decoupled by design, specifically so that a failure in management infrastructure does not take down the workloads depending on it. It is one of the architectural advantages that makes OpenStack viable for large scale production environments in the first place, and it is exactly what held up here.

That guarantee only helps you if you know it exists and know how to use it to shape a recovery strategy. Recognizing it early is what turned this from an open ended crisis into a scoped, solvable engineering problem within the first hour.

The recovery

With existing workloads confirmed safe, the team turned to rebuilding the control plane. Because the need to rebuild a database from scratch is such a rare edge case, standard recovery documentation covers only damaged or degraded states. So the engineer developed and validated a new procedure for that specific scenario: reconstructing the database cluster, restoring available data in a controlled sequence, then reconnecting every OpenStack service to the new database, one dependency at a time.

That last part was the bulk of the work. The relationships linking each service to the database had to be reestablished individually, and gaps between the backup’s state and the cloud’s actual state had to be reconciled by hand, including cases where a VM was running perfectly well but the database, working from stale information, reported it as stopped or in error.

It took several days of close, sustained collaboration between the Canonical engineer and the customer’s team. By the end, the cloud was fully operational, with zero workloads lost.

What this case actually shows

Two things are worth taking from this.

Backup validation deserves the same rigor as backup scheduling. This is not a hypothetical gap. Cockroach Labs’ State of Resilience 2025 survey found that 62% of organizations do not perform regular backup restoration exercises, and 71% do no failover testing at all, despite nearly all of them running some form of resiliency testing on paper. Separately, Acronis’ Q1 2026 telemetry across its disaster recovery platform found that 85% of recovery servers had RPO monitoring turned off entirely, meaning no automatic check on whether backups are even recent enough to be useful, and only 18% of backup rules were configured for monthly testing.

The pattern is consistent: having a backup policy is common, but proving that policy actually works under real conditions is not. That gap tends to stay invisible until the day it matters, which is exactly why a tested, timed, end to end restore process belongs on a resilience checklist next to RTO and RPO targets, not treated as a given once the backup job shows green.

In a system this interconnected, architectural knowledge is what makes recovery possible at all. Knowing that OpenStack’s control and data planes are decoupled was not trivial here. It was the fact that determined the entire recovery strategy, because it told the engineer, within the first hour, that the business had not already lost. That is the kind of knowledge that matters exactly once, at the highest stakes moment, and it is precisely what deep, specialized support exists to provide on demand.

Large, distributed infrastructure accumulates operational complexity and interdependency by nature, at every vendor and every layer of the stack. What determines the outcome when something goes wrong is not whether an environment is complex. It is whether the team responding has the depth to move fast, diagnose correctly, and build a solution under pressure when the documented path runs out.

The procedure developed during this incident has since been documented for future cases. But the case itself is a useful reminder of what a support subscription is actually buying: not just answers to known problems, but access to the expertise to navigate the ones nobody has written down yet.

If your organization runs OpenStack in production, talk to us about Ubuntu Pro with Support.