Skip to main content

Rollback

Rollback restores the previous working control-plane version after an upgrade failure. First identify whether the issue comes from Chart, image, CRD, values, webhook certificate, or external dependencies.

Before Rollback​

Check Helm revisions:

helm history testudo -n disaster-system

Inspect current state:

kubectl -n disaster-system get pods
kubectl -n disaster-system describe pod <pod-name>
kubectl -n disaster-system get events --sort-by=.lastTimestamp

If the issue is object storage, remote cluster reachability, image registry, or License, do not start with a Chart rollback.

Helm Rollback​

helm rollback testudo <REVISION> -n disaster-system

Watch rollout:

kubectl -n disaster-system rollout status deploy/disaster-server
kubectl -n disaster-system rollout status deploy/disaster-web
kubectl -n disaster-system rollout status deploy/disaster-operator-controller-manager

CRD Rollback Notes​

Helm rollback may not safely roll back CRDs. If a new version already wrote fields unknown to the old controller, rollback can cause:

  • Old controller ignores new fields.
  • Status parsing fails.
  • Webhook validation does not match.

CRD-changing releases must provide migration and rollback guidance. Without explicit guidance, do not automatically roll back CRDs.

Post-rollback Verification​

kubectl -n disaster-system get disasterinstances.testudo.softcdata.com
kubectl -n disaster-system get disasteroperation
kubectl -n disaster-system get appbackups,apprestores

In the console, verify:

  • Login works.
  • Instance list and detail pages work.
  • Watch streams work.
  • Backup/restore/disaster recovery history is readable.

When Not To Roll Back​

  • A failover, reprotect, undo, cancel, or drill cleanup is running.
  • A CRD data migration has completed and the old version is incompatible.
  • The problem is remote cluster or object storage failure.
  • The problem is a business workload failure.

Pause new operations, preserve evidence, and follow release notes for manual recovery.