How to Roll Back a Bad Deploy Without Making It Worse

The rollback button is the most confidently described and least frequently tested part of most deployment setups. Teams will tell you they can roll back in seconds. Ask when they last did it, on purpose, in production, and the answer is usually “we have not needed to”.

Then a Friday afternoon deploy takes checkout down, someone hits rollback, and the old code starts throwing 500s because release N dropped a column that release N minus 1 still queries. Now you have two outages: the original bug and a rollback that failed halfway.

Rollback is not an undo button. It is a claim about the compatibility between your last two releases, and that claim is only true if you designed for it.

Roll Back First, Diagnose Second

The instinct after a bad deploy is to work out what broke. Resist it. Diagnosis is unbounded work with users watching. Rollback is bounded work with a known outcome.

The order that keeps outages short is: restore service, then investigate the artefact you just removed, at your own pace, with logs and traces you can read without a manager in the channel.

That said, rollback is not always the right move. The decision comes down to whether anything irreversible has happened since the deploy.

Situation	Action	Why
Application bug, no schema change	Roll back	Previous artefact is known good
Performance regression under load	Roll back	Restores a known baseline instantly
Additive migration only (new nullable column)	Roll back	Old code ignores the new column
Destructive migration already applied	Roll forward	Old code cannot run against the new schema
Data backfill already written	Roll forward	Rollback does not undo the data
New message format already consumed	Roll forward	Consumers have committed offsets
Third-party or config change outside the artefact	Neither	Revert the config, not the release

The pattern in the right-hand column: rollback works when the only thing that changed was code. The moment state changed, going backwards stops being free.

The Four Things That Break Rollbacks

1. Schema changes that destroy structure

This is the big one. Dropping a column, renaming a table, adding a NOT NULL constraint, or narrowing a type all make the previous release invalid. The rollback deploys cleanly and then fails on the first query.

The fix is expand and contract, what Martin Fowler calls parallel change ↗. You split one breaking change across three releases:

The cost is three deploys instead of one. The benefit is that for the whole window between release 1 and release 3, you can go backwards freely. Our guide to database migrations without the fear covers the mechanics of each phase.

2. Caches holding the new format

Release N starts serialising a session object with a new field layout and writes it to Redis. You roll back. Release N minus 1 reads those entries, fails to deserialise, and throws on every request from any user who was active during the bad window.

Cache entries outlive deploys. Either version your cache keys so a rollback reads a different namespace, or make deserialisation tolerant of unknown and missing fields. Silent tolerance beats a strict parser here.

3. Queues and event consumers

The same problem with worse timing. If release N publishes messages in a new shape, rolling back the publisher does not recall the messages already in flight. The old consumer picks up a message it cannot parse and either crashes or dead-letters it.

Publishers should be rolled back before consumers, and consumers should accept both shapes for at least one release. This is the same idempotency and compatibility discipline that makes idempotent API endpoints safe to retry.

4. Client-side assets already in browsers

A single page app deploy leaves users holding the old JavaScript bundle, which requests API endpoints that may have changed. Rolling back the server while clients hold the new bundle produces a version mismatch that looks like random errors. Keep old asset versions available rather than purging them on deploy, and give the client a way to detect a version skew and reload.

Make the Rollback Path Boring

A rollback that requires thought during an incident is a rollback that will not happen. Three properties make it dependable.

Immutable, addressable artefacts. The previous release must exist as a built artefact you can redeploy by digest, not a commit you rebuild from source. If rollback triggers a fresh CI run, your recovery time is your build time. This is one of the design constraints worth baking into your CI/CD pipeline from the start.

One command, one operator. Not a sequence of five steps across two dashboards. Whoever is on call should be able to run it without asking anyone.

Automatic triggers where the signal is clean. If error rate exceeds a threshold within ten minutes of a deploy, roll back without a human in the loop. This only works if your alerting is trustworthy, which is a question of observability rather than monitoring: you need to distinguish a genuine regression from a noisy dependency.

Reduce What You Need to Roll Back

The best rollback is one that only affects a fraction of users. Two techniques do most of the work.

Progressive delivery. Route a small slice of traffic to the new release and compare it against the old one before going wide. Google’s SRE workbook chapter on canarying releases ↗ is the clearest treatment of choosing the canary population and the evaluation window. The related tradeoffs across blue-green, canary and rolling updates are covered in our post on deployment strategies.

Decouple release from deploy. Ship the code dark behind a flag, then enable it separately. Turning off a flag is faster and safer than redeploying, and it does not touch the artefact at all. Feature flags turn most rollbacks into a config change, though they come with their own cleanup debt if you never remove the dead branches.

Practise It

Pick a low-traffic window. Deploy the current release again, then roll back to the previous one, on purpose, with a stopwatch. You will find out three things: whether the command works, how long it actually takes, and whether anyone other than the person who wrote it knows how to run it.

Teams that do this quarterly recover in minutes. Teams that do not discover their rollback is broken at the worst possible moment, which is how a ten minute incident becomes a ninety minute one and eats a month of error budget.

Before the next release that touches schema, cache format, or message shape, write down one line: what happens if we roll this back an hour after deploy. If the honest answer is “we are not sure”, you have found the work worth doing this sprint.