Failover Auto-Compensation

Failover is a multi-step long-running operation. When a step fails or times out, the operator first identifies where the failure happened, then decides whether it can compensate automatically. Auto-compensation does not turn the failed DisasterOperation into a success. Its goal is to bring the instance back to a controllable protection state whenever it is safe to do so.

Auto-compensation only applies to operationType=failover. Failures in reprotect, undo, cancel, sync, or drill operations are recorded with failure details, but they do not use the failover auto-compensation path.

Trigger Conditions

During failover, each step is recorded in DisasterOperation.status.steps. Failure handling starts when:

• A step returns an error, such as unreachable cluster, restore policy validation failure, scale-down failure, or scale-up failure.
• A step exceeds timeoutMinutes. If the operation does not set a timeout, the operator tries to inherit spec.operationTimeoutMinutes from the instance.

note

The current platform UI does not expose an editable failover timeout field. timeoutMinutes is a backend action parameter/CRD field. It is written to DisasterOperation.spec.timeoutMinutes by the frontend request or by direct API calls. If the operation does not set it, the operator inherits spec.operationTimeoutMinutes from the instance, whose default is 60 minutes.

Therefore, a "step timed out" message in the console means the operator applied its backend timeout logic. It does not mean the user configured that timeout from the UI.

The failure flow is:

Failover step failed or timed out
  -> resolve autoCancel mode
  -> write autoCancel status
  -> recover instance state when possible

Compensation Modes

The current implementation has three modes:

Mode	Steps	Behavior	Instance result
DirectRollback	PreCheck	No business-changing action has started, so the instance is directly returned to Protected.	Protected
CancelPath	PauseSchedules, FinalSync, ScaleDownSource, ScaleUpTarget, CheckReplicas	Runs compensation steps: scale down target, scale up source, resume sync schedules.	Protected after successful compensation
NoAutoCancel	SwitchRoles or unknown stage	Does not compensate automatically and requires manual intervention.	Failed

Even when auto-compensation succeeds, the failover operation remains Failed, because the failover itself did not complete. To understand the real outcome, check both status.autoCancelStatus and the instance state.

DirectRollback

DirectRollback is used when PreCheck fails. At this point the operator has not paused schedules, triggered final sync, scaled down the source, or scaled up the target. No reverse action is needed.

The operator:

• Marks the failed step as Failed.
• Marks DisasterOperation.status.state as Failed.
• Sets autoCancelTriggered=true.
• Sets autoCancelStatus=Succeeded.
• Sets autoCancelMode=DirectRollback.
• Restores the instance to Protected.
• Clears instance error state.

Typical cases:

• Target cluster is unreachable.
• RestorePolicy dry-run fails.
• Modifier rule submission validation fails.
• Source cluster is unreachable and force=true was not used.

CancelPath

CancelPath is used after the failover has entered the transition path. Even a FinalSync failure may happen after schedules have been paused. A failure in ScaleDownSource, ScaleUpTarget, or CheckReplicas may leave source or target replicas changed. The operator therefore runs explicit compensation steps.

The steps are recorded in:

DisasterOperation.status.autoCancelSteps

The order is:

ScaleDownTarget
  -> ScaleUpSource
  -> ResumeSchedules

1. ScaleDownTarget

The operator scales down Deployments and StatefulSets in protected namespaces on the target cluster.

This prevents the target side from staying active. It does not delete ResourceSync-restored resource skeletons or object storage backups.

2. ScaleUpSource

The operator scales up Deployments and StatefulSets in protected namespaces on the source cluster.

It first reads the replica-count ConfigMap recorded by ResourceSync:

replicas-<resourceSyncName>

If that record is unavailable, it tries to read the workload annotation:

testudo.softcdata.com/original-replicas

ScaleUpSource in the CancelPath waits for workloads to become Ready by default. If the source cluster is unreachable or workloads cannot become Ready, auto-compensation fails and requires manual intervention.

3. ResumeSchedules

The operator resumes DataSync and ResourceSync schedules:

DataSync.spec.paused = false
ResourceSync.spec.paused = false

After successful compensation, the instance returns to Protected and keeps the original primary/secondary direction.

NoAutoCancel

The operator does not auto-compensate after SwitchRoles failure or an unknown failure stage.

At that point the operation is at the role boundary. The system cannot safely infer whether external traffic, source replicas, target replicas, and instance roles are all aligned. Automatic reverse action may make the situation worse.

The operation records:

autoCancelTriggered=false
autoCancelStatus=NotTriggered
autoCancelMode=NoAutoCancel
manualInterventionRequired=true

The instance enters Failed. Operators must inspect the environment and decide whether to repair, reset, retry failover, or run a manual rollback process.

Status Fields

The raw CR exposes:

status:
  state: Failed
  currentStep: FinalSync
  message: Failover was auto-compensated after FinalSync failed and the instance returned to Protected
  autoCancelTriggered: true
  autoCancelStatus: Succeeded
  autoCancelMode: CancelPath
  autoCancelReason: step timed out...
  autoCancelTriggerStep: FinalSync
  autoCancelCurrentStep: ""
  autoCancelSteps:
    - name: ScaleDownTarget
      state: Completed
    - name: ScaleUpSource
      state: Completed
    - name: ResumeSchedules
      state: Completed
  autoCancelTriggeredAt: "2026-05-15T..."
  autoCancelCompletionTime: "2026-05-15T..."
  manualInterventionRequired: false

The server summarizes this on instance lists, instance details, operation details, and history records:

{
  "autoCancel": {
    "triggered": true,
    "status": "Succeeded",
    "reason": "step timed out...",
    "triggerStep": "FinalSync",
    "manualInterventionRequired": false,
    "triggeredAt": "2026-05-15T15:01:00+08:00",
    "completionTime": "2026-05-15T15:02:00+08:00"
  }
}

Interpret the result as follows:

Operation state	Auto-compensation state	Instance state	Meaning
Failed	Succeeded	Protected	Failover failed, but the system recovered the instance to the original protection direction.
Running	Running	Usually FailingOver	Auto-compensation is still running.
Failed	Failed	Failed	Failover failed and compensation also failed. Manual intervention is required.
Failed	NotTriggered	Failed	This stage does not support auto-compensation. Manual intervention is required.

Troubleshooting

List the latest failover operation:

kubectl -n disaster-system get disasteroperation \
  -l testudo.softcdata.com/instance=<instance-name>

Inspect auto-compensation fields:

kubectl -n disaster-system get disasteroperation <operation-name> -o yaml

Check:

• status.currentStep
• status.steps
• status.message
• status.autoCancelTriggered
• status.autoCancelStatus
• status.autoCancelMode
• status.autoCancelSteps
• status.manualInterventionRequired

Also inspect the instance:

kubectl -n disaster-system get disasterinstance <instance-name> -o yaml

If autoCancelStatus=Succeeded, the instance should be back to Protected. It can continue scheduled protection or run another failover later. If manualInterventionRequired=true, do not trigger failover again immediately. First check:

• Whether source Deployments/StatefulSets are running.
• Whether target Deployments/StatefulSets are scaled back to 0.
• Whether DataSync and ResourceSync are still paused=true.
• Whether DNS, Ingress, gateway, or load balancer traffic was already switched manually.
• Whether object storage, Velero Backup/Restore, or PodVolumeRestore has failed events.

Boundaries

• Auto-compensation does not replace external traffic switching or rollback. DNS, GSLB, gateways, and business traffic must still follow the runbook.
• Auto-compensation does not delete every restored object on the target side. It scales down target workloads, scales up source workloads, and resumes sync schedules.
• If production traffic was already switched to the target side, humans must still verify traffic direction after compensation.
• If a compensation step fails, the instance enters Failed and manualInterventionRequired=true is recorded.
• SwitchRoles failure does not trigger auto-compensation because the system is already at the role boundary.
• If production failover is triggered through the API or CRD, use a realistic timeoutMinutes value based on data volume and workload size. Overly short timeouts can interrupt final sync or readiness checks. When failover is triggered from the current console, the UI does not provide a manual field for this value.

Relationship With Manual Cancel

Auto-compensation reuses the core cancel path steps, but it is not a user-created operationType=cancel.

• Auto-compensation: triggered inside the failed failover operation and recorded in the same DisasterOperation.status.autoCancel* fields.
• Manual cancel: explicitly created by the user as operationType=cancel to stop or recover a running or failed transition.

After successful auto-compensation, manual cancel is usually unnecessary. If auto-compensation fails or is not triggered, decide whether to run manual cancel, reset, or repair the environment based on the actual cluster state.