Operator Runtime Configuration

OperatorRuntimeConfig dynamically tunes disaster-operator runtime controls. It covers timeout, polling, retry, requeue, and watchdog behavior for backup, restore, disaster operations, disaster instances, data sync, resource sync, storage repositories, and cluster management controllers.

These settings are for platform administrators. They do not replace explicit settings on individual business resources, such as the timeout of a single backup, restore, or failover operation. They provide platform defaults and controller runtime controls.

Configuration Object

The operator reads one singleton object in the management namespace:

apiVersion: testudo.softcdata.com/v1
kind: OperatorRuntimeConfig
metadata:
  name: default
  namespace: disaster-system
spec: {}

Requirements:

• metadata.name must be default.
• metadata.namespace must be the operator management namespace. The default is disaster-system.
• OperatorRuntimeConfig objects with other names or in other namespaces do not affect runtime behavior.

Precedence

Runtime settings are resolved in this order:

1. Resource-level spec, such as AppBackup.spec.timeout, AppRestore.spec.timeout, DisasterOperation.spec.timeoutMinutes, and DisasterOperation.spec.retryPolicy.
2. Instance or policy defaults, such as DisasterInstance.spec.operationTimeoutMinutes.
3. Hot runtime settings from OperatorRuntimeConfig/default.
4. Operator startup environment variables or flags, including compatible APPRESTORE_* environment variables.
5. Built-in operator defaults.

Hot Reload Behavior

After OperatorRuntimeConfig/default changes, the operator creates a new runtime snapshot during the next reconcile of that object. Running workflows are not forcibly interrupted, but later reconciles, timeout checks, and requeue calculations read the latest snapshot.

Hot reload is not real-time. There can be delay between the Kubernetes watch/cache update and the next reconcile of a specific controller.

If OperatorRuntimeConfig/default is deleted, the operator falls back to startup defaults and built-in defaults. If the object exists but contains invalid values, the operator keeps the last valid snapshot and writes Invalid=True to the object status.

Check current status:

kubectl get operatorruntimeconfig default -n disaster-system
kubectl describe operatorruntimeconfig default -n disaster-system

Validation And Status

The CRD schema validates object shape and basic types. Runtime ranges and cross-field relationships are validated by the operator.

For invalid configuration:

• Kubernetes API server can persist the object.
• The operator does not activate that generation.
• status.conditions shows Ready=False and Invalid=True.
• status.activeGeneration remains the last successfully activated generation.

Common invalid values:

• backupRuntime.pollInterval: 0s
• operationRuntime.defaultTimeoutMinutes: 0
• syncRuntime.historyRetention: 0
• instanceRuntime.transitionWatchdogTimeout smaller than instanceRuntime.minTransitionWatchdogTimeout

Duration values use Kubernetes duration syntax, such as 5s, 90s, 10m, and 2h.

Full Example

apiVersion: testudo.softcdata.com/v1
kind: OperatorRuntimeConfig
metadata:
  name: default
  namespace: disaster-system
spec:
  backupRuntime:
    inProgressMaxWait: 2h
    unknownMaxWait: 10m
    pollInterval: 10s
  restoreRuntime:
    inProgressMaxWait: 1h
    unknownMaxWait: 1h
    inProgressPollInterval: 5s
    unknownPollInterval: 10s
    progressCompleteGrace: 5m
    startupGrace: 5m
    missingGrace: 90s
    emptyStatusGrace: 5m
    podVolumeRestorePendingMaxWait: 10m
    retryBackoff: 15s
    retryLimit: 1
    retryLimitProgress: 1
    retryLimitStartup: 1
    retryLimitMissing: 2
    retryLimitEmpty: 2
  operationRuntime:
    defaultTimeoutMinutes: 60
    stepStartRequeue: 1s
    stepRunningRequeue: 5s
    defaultRetryInterval: 5s
  instanceRuntime:
    transitionWatchdogTimeout: 2m
    minTransitionWatchdogTimeout: 30s
    initializingRequeue: 10s
    steadyRequeue: 60s
    failedRequeue: 60s
  syncRuntime:
    schedulerUpdateTimeout: 30s
    backupObserveRequeue: 2s
    backupInProgressRequeue: 5s
    historyMissingRequeue: 5s
    restoreObserveRequeue: 10s
    historyRetention: 20
  storageRuntime:
    requeueInterval: 10s
  clusterRuntime:
    reconcileInterval: 1m
    deletionRetryInterval: 10s
    veleroInstallTimeout: 10m
    veleroZombieLockThreshold: 10m

backupRuntime

backupRuntime affects how the AppBackup controller observes Velero Backup status, detects timeouts, and polls running backups.

Setting	Default	Valid Range	Purpose
backupRuntime.inProgressMaxWait	2h	1m to 24h	Maximum time a Velero Backup may stay running. After this, the operator treats the backup as timed out, tries to terminate it, and marks the AppBackup history record failed.
backupRuntime.unknownMaxWait	10m	1m to 24h	Maximum time a Velero Backup may stay with an empty or unknown phase. This handles cases where the Backup object exists but status does not reach a clear phase.
backupRuntime.pollInterval	10s	1s to 5m	Requeue interval for checking Velero Backup status while the latest backup is still in progress.

If AppBackup.spec.timeout is set, it overrides inProgressMaxWait and unknownMaxWait. A very small pollInterval increases API server load. A very small inProgressMaxWait can mark large backups as timed out too early.

restoreRuntime

restoreRuntime affects how the AppRestore controller handles Velero Restore timeouts, stall detection, automatic retry, and observation intervals.

Setting	Default	Valid Range	Purpose
restoreRuntime.inProgressMaxWait	1h	1m to 24h	Maximum time a Velero Restore may stay InProgress. After this, the operator treats the restore as timed out and may terminate the restore and clean pending restored resources.
restoreRuntime.unknownMaxWait	1h	1m to 24h	Maximum time a Velero Restore may stay with an empty or unknown phase.
restoreRuntime.inProgressPollInterval	5s	1s to 5m	Requeue interval for observing a restore while it is InProgress.
restoreRuntime.unknownPollInterval	10s	1s to 5m	Requeue interval for observing a restore while its phase is empty or unknown.
restoreRuntime.progressCompleteGrace	5m	30s to 24h	Grace period after Velero reports all items restored but the restore still remains InProgress. After this, the operator treats the restore as stalled and enters retry or failure handling.
restoreRuntime.startupGrace	5m	30s to 24h	Grace period for a newly created restore that has no start or completion status.
restoreRuntime.missingGrace	90s	30s to 24h	Grace period when the expected Velero Restore object is missing.
restoreRuntime.emptyStatusGrace	5m	30s to 24h	Grace period when a Restore object exists but status stays empty.
restoreRuntime.podVolumeRestorePendingMaxWait	10m	1m to 24h	Maximum wait for PodVolumeRestore objects that remain pending or do not make progress.
restoreRuntime.retryBackoff	15s	1s to 1h	Wait time before an automatic restore retry and the next reconcile interval after retry.
restoreRuntime.retryLimit	1	0 to 10	Default automatic retry limit. When per-stall-type limits are not set, this value is used. 0 disables automatic retry.
restoreRuntime.retryLimitProgress	1	0 to 10	Automatic retry limit for progress-completed-but-still-in-progress stalls.
restoreRuntime.retryLimitStartup	1	0 to 10	Automatic retry limit for restore startup stalls or transient Velero server-starting failures.
restoreRuntime.retryLimitMissing	2	0 to 10	Automatic retry limit when the Velero Restore object is missing.
restoreRuntime.retryLimitEmpty	2	0 to 10	Automatic retry limit when Restore status stays empty.

If AppRestore.spec.timeout is set, it overrides inProgressMaxWait and unknownMaxWait. Very small grace values can misclassify transient Velero status lag as a stall. Very small retryBackoff or poll intervals can cause frequent reconciles during failures.

operationRuntime

operationRuntime affects the DisasterOperation controller, including failover, reprotect, undo, cancel, and drill workflows.

Setting	Default	Valid Range	Purpose
operationRuntime.defaultTimeoutMinutes	60	1 to 1440	Default timeout, in minutes, for a disaster operation when DisasterOperation.spec.timeoutMinutes is not set.
operationRuntime.stepStartRequeue	1s	1s to 5m	Requeue interval after an operation step is created or started.
operationRuntime.stepRunningRequeue	5s	1s to 5m	Requeue interval while an operation step is running.
operationRuntime.defaultRetryInterval	5s	1s to 1h	Default wait before operation retry when DisasterOperation.spec.retryPolicy.retryIntervalSeconds is unset or not positive.

DisasterOperation.spec.timeoutMinutes takes precedence over defaultTimeoutMinutes. DisasterOperation.spec.retryPolicy.retryIntervalSeconds takes precedence over defaultRetryInterval. Small step requeue values increase Kubernetes API usage.

instanceRuntime

instanceRuntime affects how the DisasterInstance controller handles state transitions, no-progress watchdog checks, and failed-state rechecks.

Setting	Default	Valid Range	Purpose
instanceRuntime.transitionWatchdogTimeout	2m	30s to 24h	Timeout for detecting that a disaster instance has stayed in a transition without observed operation progress.
instanceRuntime.minTransitionWatchdogTimeout	30s	10s to 1h	Lower bound for the watchdog timeout, even if an instance-level timeout or runtime setting is smaller.
instanceRuntime.initializingRequeue	10s	1s to 10m	Requeue interval while a DisasterInstance is initializing.
instanceRuntime.steadyRequeue	60s	5s to 30m	Regular recheck interval while a DisasterInstance is steady.
instanceRuntime.failedRequeue	60s	5s to 30m	Recheck interval while a DisasterInstance is failed.

If DisasterInstance.spec.operationTimeoutMinutes is greater than 0, the watchdog timeout prefers that value converted to minutes. The final watchdog timeout is still protected by minTransitionWatchdogTimeout. transitionWatchdogTimeout must be greater than or equal to minTransitionWatchdogTimeout.

syncRuntime

syncRuntime affects both DataSync and ResourceSync controllers. It controls scheduler update timeout, backup observation, restore observation, and history retention.

Setting	Default	Valid Range	Purpose
syncRuntime.schedulerUpdateTimeout	30s	1s to 10m	Context timeout for updating underlying scheduler configuration. If exceeded, the update fails and waits for a later reconcile.
syncRuntime.backupObserveRequeue	2s	1s to 5m	Requeue interval after creating or triggering backup while waiting to observe backup objects and history.
syncRuntime.backupInProgressRequeue	5s	1s to 5m	Requeue interval while the underlying backup is still in progress.
syncRuntime.historyMissingRequeue	5s	1s to 5m	Requeue interval while expected backup or restore history has not yet appeared in status.
syncRuntime.restoreObserveRequeue	10s	1s to 5m	Requeue interval after triggering restore while waiting to observe restore status and history.
syncRuntime.historyRetention	20	1 to 500	Maximum number of entries retained in DataSync and ResourceSync status history. Older entries are trimmed.

A small historyRetention reduces troubleshooting history. Small observe and requeue values make state changes visible sooner but increase Kubernetes API load.

storageRuntime

storageRuntime affects the StorageRepository controller when checking object storage connectivity, capacity statistics, and BackupStorageLocation-related status.

Setting	Default	Valid Range	Purpose
storageRuntime.requeueInterval	10s	5s to 1h	Requeue interval after each StorageRepository validation, status sync, or failure retry.

A small value increases object-storage API, Kubernetes API, and controller load. A large value slows visibility of storage changes, connectivity recovery, and capacity refreshes.

clusterRuntime

clusterRuntime affects the Cluster controller when reconciling remote cluster connectivity, Velero installation, and deletion.

Setting	Default	Valid Range	Purpose
clusterRuntime.reconcileInterval	1m	10s to 1h	Regular Cluster controller state-check and reconcile interval.
clusterRuntime.deletionRetryInterval	10s	1s to 10m	Retry interval during Cluster deletion when remote cleanup or finalizer handling has not completed.
clusterRuntime.veleroInstallTimeout	10m	1m to 2h	Value passed as --timeout when the operator runs Velero Helm install or upgrade. It only affects future commands, not an already running Helm command.
clusterRuntime.veleroZombieLockThreshold	10m	5m to 24h	Threshold for detecting long-running Velero Helm release locks or in-progress states.

Tune veleroInstallTimeout according to image pull speed, node count, and network conditions. A small veleroZombieLockThreshold can misclassify normal Helm operations, while a large value delays detection of stuck installs or upgrades.

Recommendations

• Do not lower all requeue, poll, and retry backoff settings at once. Frequent reconciles across multiple controllers amplify management API server load.
• Tune timeout and grace settings based on real data size, image pull speed, cross-cluster network, and Velero plugin performance.
• High automatic retry limits can hide real configuration errors and keep consuming cluster resources.
• Record the current configuration and status before changing runtime settings.
• Validate changes in a test environment first, then adjust one group at a time in production.

Rollback

Return to startup defaults:

kubectl delete operatorruntimeconfig default -n disaster-system

Return to a previous YAML:

kubectl apply -f operator-runtime-config-backup.yaml

Confirm activation:

kubectl get operatorruntimeconfig default -n disaster-system \
  -o jsonpath='{.status.activeGeneration}{"\n"}{.status.conditions}{"\n"}'