Disaster Recovery Troubleshooting
Follow the resource chain: DisasterInstance creates DataSync and ResourceSync; user actions create DisasterOperation; drills create DisasterDrill.
Quick Map
| Symptom | Check First |
|---|---|
Instance stuck in Pending | Configuration, clusters, repository, policies |
Instance stuck in Initializing | First DataSync or ResourceSync |
ConfigError | Namespaces, policies, restore policy, cluster references |
| DataSync failed | AppBackup/AppRestore, Velero, object storage |
| ResourceSync failed | Target CRDs, resource conflicts, modifiers |
| Failover failed | Operation steps, target replicas, final sync |
Instance Failed with reason=RoleDriftDetected | Expected primary/secondary roles and Deployment/StatefulSet replicas in both downstream clusters |
| Drill failed | Backup availability, target namespace, cleanup leftovers |
If the message contains expectedPrimary=... role=Standby ... expectedSecondary=... role=Standby ... both clusters are scaled to zero, both downstream clusters were sampled with zero replicas. See Role Drift And Replica Consistency Check for the decision logic and repair steps.
Commands
kubectl get disasterinstance,datasync,resourcesync,disasteroperation,disasterdrill -A
kubectl get events -A --sort-by=.lastTimestamp | tail -100
kubectl -n disaster-system logs deploy/disaster-operator --tail=300
For one instance:
kubectl get disasterinstance <name> -o yaml
kubectl get datasync,resourcesync -l testudo.softcdata.com/instance=<name>
kubectl get disasteroperation -l testudo.softcdata.com/instance=<name>
Do not hand-edit status. Keep failed operations and events until diagnosis is complete.