Skip to main content

Upgrade

Upgrade Testudo as one release unit: Chart, CRDs, operator, server, web, and values files. Avoid upgrading only one component.

Pre-checks​

helm ls -n disaster-system
kubectl -n disaster-system get pods
kubectl -n disaster-system get disasteroperation
kubectl -n disaster-system get appbackups,apprestores

Confirm:

  • No failover, reprotect, undo, cancel, or drill cleanup is running.
  • No critical backup or restore task is running.
  • Current values are backed up.
  • Current Chart package, image tags, and CRD versions are recorded.
  • The new compatibility matrix has been reviewed.

Back Up Current State​

helm get values testudo -n disaster-system -o yaml > values-current.yaml
helm get manifest testudo -n disaster-system > manifest-current.yaml

kubectl get crd | grep testudo.softcdata.com
kubectl -n disaster-system get disasterinstances.testudo.softcdata.com -o yaml > disasterinstances-backup.yaml
kubectl -n disaster-system get disastergroups.testudo.softcdata.com -o yaml > disastergroups-backup.yaml

Do not commit exported values or CRDs containing credentials to a public repository.

Dry Run​

helm template testudo ./testudo-chart-1.0.0.tgz \
  -n disaster-system \
  -f values-prod.yaml > rendered-new.yaml

helm upgrade testudo ./testudo-chart-1.0.0.tgz \
  -n disaster-system \
  -f values-prod.yaml \
  --dry-run

Check:

  • CRD changes.
  • Webhook changes.
  • Service port changes.
  • Image tags.
  • global.namespace and release namespace alignment.

Apply CRD Schema​

Helm treats the chart crds/ directory specially: CRDs are created during installation, but helm upgrade does not update CRDs that already exist in the cluster. Whenever a release includes CRD schema changes, explicitly apply the CRD manifests from the chart package.

If the release changes only CRDs, do not run helm upgrade. Apply the CRDs and verify them:

mkdir -p /tmp/testudo-chart-crds
tar -xzf ./testudo-chart-1.0.0.tgz -C /tmp/testudo-chart-crds
CHART_DIR=/tmp/testudo-chart-crds/testudo-chart

kubectl apply --server-side --force-conflicts -f "$CHART_DIR/crds/operator-crds.yaml"
kubectl apply --server-side --force-conflicts -f "$CHART_DIR/crds/velero-crds.yaml"

kubectl wait --for condition=Established --timeout=120s crd \
  appbackups.testudo.softcdata.com \
  apprestores.testudo.softcdata.com \
  disasterinstances.testudo.softcdata.com \
  disasterconfigs.testudo.softcdata.com \
  resourcesyncs.testudo.softcdata.com \
  backups.velero.io \
  restores.velero.io \
  schedules.velero.io

If the release changes both CRDs and runtime resources, apply CRDs first, wait until they are Established=True, and then run helm upgrade. This prevents the new controller or webhook from using fields while the API server still validates objects with the old schema.

Run Upgrade​

helm upgrade testudo ./testudo-chart-1.0.0.tgz \
  -n disaster-system \
  -f values-prod.yaml

Watch rollout:

kubectl -n disaster-system rollout status deploy/disaster-server
kubectl -n disaster-system rollout status deploy/disaster-web
kubectl -n disaster-system rollout status deploy/disaster-operator-controller-manager
kubectl -n disaster-system get events --sort-by=.lastTimestamp

Post-upgrade Verification​

kubectl api-resources | grep testudo.softcdata.com
kubectl get crd disasterinstances.testudo.softcdata.com -o yaml | grep -E "rewriteImage|sourcePrefix|targetPrefix" || true
kubectl -n disaster-system get pods,svc
curl -I http://<NodeIP>:30087

In the console, verify:

  • Dashboard statistics load.
  • Cluster, storage, and instance lists load.
  • Existing DisasterInstance states are unchanged.
  • Watch streams work.
  • A small backup or drill works in a test namespace.

Risks And Limits​

  • Do not upgrade during long-running operations.
  • CRD schema changes require release-note review, possible migration, and an explicit kubectl apply of the CRDs from the chart package.
  • Webhook certificate or CA bundle changes can break CRD writes.
  • If server authentication changes, validate login, refresh token, Watch API, and OpenAPI.