Designing Break-Glass That Survives a Cloud Outage
Glassbreak Team · Published 2026-07-17
If your break-glass procedure lives on the same cloud provider as the systems it's meant to recover, it isn't a recovery path — it's a second thing that can go down in the same outage. This is the single most common design flaw in emergency-access planning: teams build a careful process for triggering, approving, and logging emergency access, then host the tool that runs that process on the same infrastructure whose failure the process exists to survive.
The failure mode nobody plans for
Most break-glass planning focuses on the human process — who can trigger access, how many approvers are required, how long the grant lasts. That's necessary, but it assumes the tooling itself is reachable when you need it. In practice, the scenarios that actually trigger break-glass are disproportionately the scenarios where a shared dependency has failed: a regional AWS event that takes out your primary database and also takes out the internal admin panel you'd normally use to fix it; an identity provider outage that locks you out of production and locks you out of the tool you'd use to recover a shared credential, because both sit behind the same SSO.
A single cloud account is a single failure domain in more ways than "the servers are down." A control-plane incident — IAM misbehaving, an API gateway rate-limiting everything, a billing hold freezing an account — can make an entire provider's surface unreachable even while individual compute instances are technically running. Multi-region deployment inside one provider doesn't fix this, because regions in the same account typically share the control plane that failed.
What actual independence requires
Surviving a cloud outage means checking independence at every layer your break-glass path touches, not just the compute layer:
- Compute. Is the emergency-access tool running on hardware in a different cloud provider than production — not just a different region of the same one?
- Database. Does the tool's data live in a database hosted by that same second provider, or does it quietly call back to the primary cloud's managed database service?
- DNS. Can you resolve the tool's hostname if your primary provider's DNS is unreachable? Shared DNS is an easy layer to miss because it doesn't look like "infrastructure" the way a server does.
- Edge / CDN. If a CDN fronts the tool, does it have a direct-to-origin fallback that bypasses the CDN entirely, or does a CDN outage take the tool down along with everything else behind it?
Missing any one of these re-creates the single point of failure one layer down. A break-glass tool that runs on a second cloud but stores its data in the first cloud's managed Postgres has solved the compute problem and reintroduced the database problem.
How this actually gets designed
Glassbreak's approach is to run genuinely independent compute stacks rather than one deployment with redundancy features bolted on. Today that means two always-on boxes — one on AWS, one on Scaleway — each running the identical application image against its own in-box PostgreSQL, with Microsoft Azure planned as a third. Neither stack calls back to the other's database, and neither depends on the other cloud's control plane for anything on the request path. The two stacks are kept consistent with native Postgres streaming replication rather than an application-level sync layer, and health-checked failover routes traffic to whichever stack is actually healthy. You can read the full layer-by-layer breakdown — registrar, DNS, edge, compute, database — on the distributed infrastructure page.
The practical effect for a team using Glassbreak as their break-glass layer: a whole-provider outage at your primary cloud, even a severe one that takes out the control plane, doesn't take Glassbreak down with it, because Glassbreak was never depending on that provider to begin with.
Building this into your own evaluation
If you're evaluating a break-glass or emergency-access tool, ask the vendor the four-layer question above directly — compute, database, DNS, edge — rather than accepting "highly available" as an answer, since high availability inside one cloud and independence from that cloud are different guarantees. And test it: during your next tabletop drill, simulate your primary provider being fully unreachable and confirm the emergency-access tool is still reachable, not just that the process document says it should be. For the broader set of practices a resilient break-glass design should follow beyond infrastructure independence — quorum approval, time-boxed grants, a real audit trail — see break-glass account best practices. If you're ready to see the model in action, start for free with a single team before committing to a full rollout.
Frequently asked questions
- Isn't multi-region inside one cloud provider enough?
- It covers regional failures but not control-plane failures. Multi-region AWS deployments, for example, still share the IAM control plane, the API gateway layer, and often the account itself — an incident there can take out every region at once. True independence means a different provider, not just a different region of the same one.
- How do I know if my break-glass tool actually depends on my primary cloud?
- Ask where it runs, where its database lives, and whose DNS or CDN fronts it. If any of those three match your production stack's provider, an outage that takes down production can take down your recovery path with it. The test is whether you could reach the tool if your primary cloud account were completely unreachable, not just degraded.
- Does this mean I need to run my own infrastructure on two clouds myself?
- Not necessarily — that's exactly the operational burden a purpose-built break-glass platform is meant to absorb. What matters is that whatever tool holds your emergency-access path has already solved multi-cloud independence for itself, so you don't have to replicate that engineering effort just to have a recovery plan that works.