Monitoring
Testudo observability comes from CRD status, conditions, history, Kubernetes Events, Server Watch APIs, logs, and operator metrics.
Key Objects
| Object | Fields To Watch |
|---|---|
DisasterInstance | status.fsmState, reason, message, conditions |
DataSync | Latest backup/restore, state, history |
ResourceSync | phase, restore status, modifier result |
DisasterOperation | currentStep, steps, state, autoCancel |
AppBackup / AppRestore | Velero object, failure reason |
BackupRestoreStatistics | backup, restore, instance, operation statistics |
Basic Checks
kubectl -n disaster-system get disasterinstances.testudo.softcdata.com
kubectl -n disaster-system get disasteroperation
kubectl -n disaster-system get appbackups,apprestores
kubectl -n disaster-system get events --sort-by=.lastTimestamp
Logs
kubectl -n disaster-system logs deploy/disaster-operator-controller-manager
kubectl -n disaster-system logs deploy/disaster-server
kubectl -n disaster-system logs deploy/disaster-web
Send operator and server logs to centralized logging, and index by instance name, operation name, and trace ID.
Metrics
The operator exposes controller-runtime metrics. The current deployment uses metrics service port 8443 and protects /metrics with authentication and authorization.
Verify:
kubectl -n disaster-system get svc | grep metrics
kubectl -n disaster-system get endpoints | grep metrics
When integrating Prometheus, ensure:
- Prometheus ServiceAccount has metrics access RBAC.
- TLS and bearer token are configured.
- NetworkPolicy allows the monitoring namespace to reach the metrics service.
Alert Suggestions
Cover at least:
- Instance stuck before
Protected. DisasterOperationstuck on one step.- AppBackup/AppRestore failures.
- Velero Backup/Restore failures.
- BSL unavailable.
- Server 5xx or authentication errors increase.
- Object storage capacity or error-rate abnormal.
Console And Watch API
The console gets state through REST/Watch. If pages stop refreshing while CRD status changes normally, check server logs, Watch routes, gateway long connections, and browser Network.