Rollback Is a Feature: Designing CI/CD Pipelines That Can Recover

Delivery is incomplete without recovery

A pipeline that can deploy but cannot reliably recover is only half automated. When a release fails, responders need to identify exactly what changed, select a known artifact, determine whether old code can run against the current database, shift traffic safely, and prove that service health recovered. If those steps depend on reconstructing commands from chat history, rollback is not a feature of the system.

Design rollback alongside deployment. The relevant unit is not just an application image. It is a release composed of artifact digests, configuration, infrastructure revision, database state assumptions, feature flags, and routing state. Recovery must account for all of them.

Make artifacts immutable and promotable

Build once, then promote the same artifact through environments. Rebuilding from the same Git commit can produce a different result because base images, package indexes, build tools, or remote dependencies changed. Mutable tags such as latest or an environment name cannot establish identity.

docker buildx build \
  --tag registry.example.com/payments:${GIT_SHA} \
  --provenance=true \
  --sbom=true \
  --push .

docker buildx imagetools inspect \
  registry.example.com/payments:${GIT_SHA}

Record and deploy the registry digest, such as sha256:..., rather than trusting the tag. Prevent tag mutation in the registry. Retain artifacts for at least the operational rollback window, and replicate or escrow artifacts if registry availability is itself a recovery dependency.

The same principle applies to Helm charts, serverless bundles, VM images, static assets, and infrastructure modules. Signatures and provenance can improve supply-chain assurance, but they do not replace the core requirement: the pipeline must be able to retrieve the exact bytes that previously ran.

Keep an authoritative deployment record

Git history answers what was intended. A deployment record answers what actually ran. Each deployment should create an append-only record containing:

• Service, environment, cluster or account, and deployment identifier.
• Artifact digest and source commit.
• Configuration and secret version references, without recording secret values.
• Infrastructure and migration revisions.
• Feature-flag state or a link to an audited flag snapshot.
• Initiator, approvals, timestamps, and pipeline run URL.
• Previous release identifier and traffic strategy.
• Verification results and final status.

{
  "deployment_id": "prod-payments-20260818-142501",
  "artifact": "registry.example.com/payments@sha256:...",
  "commit": "4f7c2d1...",
  "config_revision": "payments-prod-v83",
  "db_compatibility": "schema-214-through-216",
  "previous_deployment": "prod-payments-20260811-091204"
}

Store this record somewhere responders can query when the deployment system is degraded. Kubernetes annotations are useful but should not be the only record if cluster access is part of the failure. A deployment should not be marked successful until verification evidence is attached.

Database compatibility determines whether rollback is real

Application rollback is unsafe when a migration has made the database incompatible with the previous version. Database backups solve disaster recovery, not routine release rollback. Restoring a database to an earlier point can discard valid writes made after deployment.

Use expand and contract changes across multiple releases:

1. Expand: add nullable columns, new tables, indexes, or compatible structures without removing what old code needs.
2. Migrate: deploy code that can operate during the transition. Backfill data with bounded, observable jobs.
3. Switch: move reads or writes to the new representation after validation.
4. Contract: remove old fields only after no deployable version depends on them and the rollback window has closed.

For renames, add the new field and support dual read or dual write temporarily rather than renaming destructively in one step. For constraints, validate existing data before enforcement. For large indexes, use the database's online or concurrent mechanism where available and understand its failure semantics.

Every release should declare a database compatibility range. The rollback controller can then reject an unsafe target instead of discovering incompatibility after traffic moves. Down migrations are not automatically safe. A down migration that drops data is a second incident, not a recovery.

Separate deployment from release

Blue-green deployments

Blue-green keeps the previous environment available while the new environment is verified. Traffic switching is fast, and switching back can be equally fast when session state, database compatibility, and external side effects permit it.

The hidden risks are shared dependencies and state. Both colors may write to the same database, publish events, process queues, or run scheduled jobs. Ensure only the intended color owns singleton work. Confirm that clients, caches, DNS, connection draining, and long-lived sessions respond correctly to a traffic switch.

Canary deployments

Canaries expose a new release to a bounded share of traffic or a selected cohort. Progression should be driven by service-level signals and sufficient sample volume, not elapsed time alone. A five-minute canary is weak evidence for a low-traffic endpoint or an hourly job.

Define analysis before deployment: request errors, latency, saturation, business transaction failures, queue behavior, and critical dependency calls. Compare against a suitable baseline and account for sparse data. Automatic rollback should be conservative when telemetry is absent or invalid. Missing evidence should stop promotion rather than count as success.

Retain a manual traffic control for conditions the automated analysis does not model. The pipeline should record who used it and the resulting routing state.

Use feature flags as operational controls

Feature flags can disable a risky path without replacing an otherwise healthy artifact. They are especially useful when deployment and user exposure should happen at different times. Flags are not a substitute for compatible code or a reason to skip deployment rollback.

Classify flags by purpose. Release flags should have owners and expiration dates. Operational kill switches should be tested, access-controlled, audited, and available during a pipeline outage. Permission flags and experiment flags have different safety requirements and should not share casual operational procedures.

Code must behave safely when the flag service is slow or unavailable. Decide the default per feature rather than adopting one global fail-open or fail-closed rule. Cache behavior and propagation delay belong in the rollback runbook. A switch that takes fifteen minutes to reach all processes is not an immediate recovery control.

Automate rollback without hiding judgment

A rollback workflow should take a deployment identifier, not an arbitrary image tag. It should resolve the full previous release record, check compatibility, show the proposed changes, require the appropriate approval, execute the deployment or traffic shift, and run post-rollback verification.

rollback --environment production \
  --service payments \
  --to-deployment prod-payments-20260811-091204 \
  --reason "elevated authorization failures" \
  --dry-run

The dry run should report artifact digest, configuration revision, database compatibility, routing change, migration implications, and unavailable dependencies. If the requested target is outside the compatibility window, the tool should fail closed and point to a forward-fix or disaster-recovery procedure.

Automatic rollback is appropriate when signals are fast, attributable, and reversible. It is dangerous when telemetry is noisy, the change includes irreversible side effects, or rollback would amplify load. For example, repeatedly oscillating between versions can corrupt caches or duplicate work. Add cooldowns, attempt limits, and an escalation state.

Drill the recovery path

Rollback code that has never run is an assumption. Exercise it on a schedule and after material pipeline changes. A useful drill deploys two known compatible versions, generates representative traffic, triggers the rollback path, and verifies artifact identity, routing, workload health, and data correctness.

Include less convenient scenarios over time:

• The latest release is unhealthy, but the deployment controller is available.
• The primary artifact registry is unavailable.
• The previous application version is incompatible with a completed schema contraction.
• Telemetry is missing, so automated analysis cannot decide.
• A feature flag must be disabled while the CI provider is unavailable.
• A canary produced external side effects before traffic was removed.

Do not run an unbounded failure experiment in production. Start in a production-like environment, then use tightly scoped production drills where organizational risk controls permit. The point is to test the real control plane and evidence chain, not to create drama.

Define minimum viable evidence

A green pipeline stage is not proof of recovery. Before declaring rollback complete, retain evidence that answers five questions:

1. What is running? Runtime artifact digest matches the selected deployment record.
2. Where is it running? All intended regions, clusters, or instances reached the desired revision.
3. What traffic reaches it? Router, load balancer, or service-mesh state matches the rollback plan.
4. Can it serve? Readiness plus representative synthetic or functional checks pass.
5. Did service recover? Existing service-level indicators returned within agreed operational bounds for a defined observation period.

Also verify that the failed release is no longer receiving asynchronous work, running scheduled jobs, or serving from a second region. Check database errors and queue consumers, not only HTTP traffic. Record evidence with timestamps and links to durable telemetry.

Rollback is complete when the system is healthy on a known release and the evidence supports that conclusion. It is not complete when a command returns zero.

Design the pipeline around recoverability

A recoverable pipeline has a few non-negotiable properties:

• Artifacts are immutable, identifiable by digest, retained, and promotable.
• Every deployment creates an authoritative release record.
• Database changes preserve compatibility through a declared rollback window.
• Progressive delivery limits exposure and has explicit stop conditions.
• Feature flags provide audited controls for risky behavior.
• Rollback resolves a complete release, checks safety, and verifies the result.
• Drills test both the mechanism and the evidence used to declare recovery.

This design changes incident response from improvisation into execution. It also improves routine delivery because teams must make artifact identity, compatibility, health criteria, and ownership explicit before production is at risk. Rollback is not an admission that deployment failed. It is evidence that the delivery system was designed for reality.