Upgrade, Rollback and Recovery#

Symptom#

An upgrade has gone wrong, or you need to establish whether one can be reversed. Typical shapes:

  • new pods will not become ready after a chart upgrade,

  • the service starts but rejects its configuration,

  • behaviour changed in a way callers did not expect,

  • a database migration ran and you need to know whether going back is safe.

Blast radius#

Depends entirely on whether the upgrade touched persistent state:

Change

Reversible?

Container image / chart version only

Yes — roll back to the previous version.

Configuration value rename or removal

Yes, but the old values file must be restored with it.

Forward database migration

Not automatically. Rolling the image back does not roll the schema back.

Data written in a new format

No. Data written by the new version may not be readable by the old one.

Establish which row you are in before rolling anything back. A rollback that leaves a migrated schema under an older binary can fail in ways that are worse than the upgrade.

Before upgrading#

  • Read the migration notes for every version you are crossing, not just the target. Storage Service documents removed and renamed chart properties per version; a value silently dropped because it was renamed is the most common upgrade failure, and it fails quietly — the setting simply stops taking effect.

  • Check version compatibility against the supported combinations for clients and services.

  • Know the current state: record the deployed chart versions and keep the values files you deployed from.

helm -n <namespace> list
helm -n <namespace> get values <release> > values-before-upgrade.yaml

Triage after a failed upgrade#

1. Are the new pods failing to start, or failing readiness?

kubectl -n <namespace> rollout status deployment/<service> --timeout=2m
kubectl -n <namespace> describe pod <new-pod> | tail -40
kubectl -n <namespace> logs <new-pod> --previous 2>/dev/null | tail -60

A pod that starts and then fails readiness is usually failing a dependency check — database, queue, or backend. A pod that exits immediately is usually a configuration rejection, and the log names the offending property.

2. Did a property change name?

A configuration value that no longer exists is not always an error; it may simply be ignored, which means the setting silently stops applying. If behaviour changed but nothing failed, compare your values against the migration notes for the versions you crossed.

3. Did a migration run?

kubectl -n <namespace> get pods --show-all 2>/dev/null | grep -i -E 'migrat|init'
kubectl -n <namespace> logs <pod> -c <init-container> | tail -40

If a migration completed, treat the schema as forward-moved and see “Rolling back” below.

Rolling back#

Image or chart only. Roll back through the same mechanism you deployed with, so the change is recorded rather than applied by hand:

helm -n <namespace> history <release>
helm -n <namespace> rollback <release> <revision>

Where deployment is GitOps-managed, revert the change in the deployment repository instead and let the pipeline reconcile — a direct helm rollback against a managed release is undone by the next deploy.

After a migration. Do not roll the binary back and hope. Confirm with the release notes whether the older version tolerates the newer schema. If it does not, the options are to fix forward, or to restore the database from a backup taken before the upgrade and roll back to that point — which loses everything written since.

Recovery#

  • Restore from backup — only for the state that is genuinely lost. Restoring a database to recover from a configuration mistake discards unrelated writes.

  • Re-deploy from source — the deployment repository is the source of truth. Reconciling from it repairs drift introduced by manual changes during an incident.

  • Reconcile manual changes — anything changed by hand during an incident is reverted by the next deploy. Fold the fixes that should persist into the deployment configuration while the incident is fresh.

Verify#

  • Rollout completes and pods pass readiness.

  • The capability that broke works, and the capabilities that were fine still do — an upgrade failure and its rollback can disturb more than the original change.

  • Deployed versions match what the deployment repository declares, with no drift left behind.