Role Drift And Replica Consistency Check

When the management-plane expected primary/secondary relationship does not match the real workload replica distribution in downstream clusters, the operator writes a RoleDrift condition. If the drift cannot be safely explained, the instance enters Failed with status.reason=RoleDriftDetected.

Typical message:

expectedPrimary=suse-2 role=Standby workloads=1 nonZeroWorkloads=0 zeroWorkloads=1 desiredReplicas=0;
expectedSecondary=local role=Standby workloads=1 nonZeroWorkloads=0 zeroWorkloads=1 desiredReplicas=0;
both clusters are scaled to zero

This means the management plane currently expects suse-2 to be the primary cluster and local to be the secondary cluster, but the operator sampled only replicas=0 workloads in both clusters. No cluster has an active business workload, so the instance is classified as BothStandby.

Where The Expected Roles Come From

RoleDrift does not infer current roles directly from DisasterConfig.spec.sourceCluster/targetCluster, because Failover and Reprotect can change the current direction.

The expected relationship comes from instance status:

expectedPrimary   = DisasterInstance.status.primaryCluster
expectedSecondary = DisasterInstance.status.secondaryCluster

Therefore expectedPrimary=suse-2 means the instance status currently expects suse-2 to run business replicas. It is not necessarily the original source cluster in the basic configuration.

How Real Downstream Roles Are Sampled

The operator connects to the clusters referenced by expectedPrimary and expectedSecondary and samples these workload types:

• Deployment
• StatefulSet

The sampling scope is limited by:

• DisasterInstance.spec.namespaces
• DisasterInstance.spec.labelSelector

For each cluster, the operator computes:

Field	Meaning
workloads	Number of sampled Deployments/StatefulSets.
nonZeroWorkloads	Workloads with spec.replicas > 0.
zeroWorkloads	Workloads with spec.replicas = 0.
desiredReplicas	Sum of sampled workload spec.replicas.

Real roles are classified from spec.replicas:

Real role	Rule
Active	At least one sampled workload has desired replicas greater than 0.
Standby	Workloads were sampled, but all desired replicas are 0.
Unknown	No workload was sampled, the downstream cluster is unreachable, or list failed.

If spec.replicas is empty, the operator treats it as the Kubernetes default value 1.

Decision Matrix

expectedPrimary real role	expectedSecondary real role	condition	Instance state	reason	Meaning
Active	Standby	RoleDrift=False	Keep steady state	ExpectedRoleMatched	Real replica distribution matches expected roles.
Standby	Active	RoleDrift=True	Failed	RoleReversed	The business side appears to be on the expected secondary cluster.
Standby	Standby	RoleDrift=True	Failed	BothStandby	Both sides are scaled to zero.
Active	Active	RoleDrift=False	Keep steady state	BothActiveObserved	Dual-active is recorded as an observation, not a hard error. A failover that skipped source scale-down can produce this.
Unknown	any	RoleDrift=Unknown	Keep steady state	CheckFailed / NoWorkloadObserved	The operator cannot judge reliably and does not fail the instance only because of this sample.
any	Unknown	RoleDrift=Unknown	Keep steady state	CheckFailed / NoWorkloadObserved	Same as above.

Why BothStandby Is An Error

In Pilot Light mode, normal protection expects:

• Current primary cluster: business workloads have non-zero replicas.
• Current secondary cluster: ResourceSync keeps a resource skeleton with workloads at replicas=0.

If both clusters are Standby, the system cannot confirm that there is any active service side. Continuing failover, reprotect, or manual sync may make the situation worse. The operator therefore:

• Writes status.conditions[type=RoleDrift].
• Sets status.fsmState=Failed.
• Sets status.reason=RoleDriftDetected.
• Clears status.availableOperations to block operations that would change runtime semantics.
• Records a Warning event named RoleDriftDetected.

The server does not recompute this conclusion. It echoes the operator-written conditions and conditionSummary.roleDrift to the console.

Related Code

Operator logic:

Location	Purpose
internal/controller/disasterinstance/role_drift.go	Samples downstream replicas, evaluates RoleDrift, and writes the condition.
evaluateRoleDrift	Reads status.primaryCluster/status.secondaryCluster and samples both clusters.
sampleClusterReplicaRole	Lists Deployment/StatefulSet in the instance namespace/label scope and classifies by spec.replicas.
guardByRoleDrift	Fails the instance when drift is not safely explainable.
handleProtected / handleActive	Call guardByRoleDrift in steady states.
handleFailed	Keeps checking reason=RoleDriftDetected; when real roles match again, it automatically recovers to Protected or Active.

Server echo logic:

Location	Purpose
internal/apis/disaster_instance/v1/types.go	Converts status.conditions into API conditions and extracts conditionSummary.roleDrift.

Troubleshooting

1. Inspect Instance Status

kubectl -n disaster-system get disasterinstance <instance-name> -o yaml

Focus on:

status:
  fsmState: Failed
  reason: RoleDriftDetected
  message: ...
  primaryCluster: suse-2
  secondaryCluster: local
  conditions:
    - type: RoleDrift
      status: "True"
      reason: BothStandby
      message: ...

2. Confirm Expected Roles

kubectl -n disaster-system get disasterinstance <instance-name> \
  -o jsonpath='{.status.primaryCluster}{"\n"}{.status.secondaryCluster}{"\n"}'

The first line is the expected primary cluster. The second line is the expected secondary cluster.

3. Check Replicas In Both Downstream Clusters

Run this against both the expected primary and expected secondary clusters:

kubectl --context <cluster-context> -n <protected-namespace> \
  get deploy,sts \
  -o custom-columns=KIND:.kind,NAME:.metadata.name,REPLICAS:.spec.replicas,READY:.status.readyReplicas

If the instance has a label selector, apply the same selector:

kubectl --context <cluster-context> -n <protected-namespace> \
  get deploy,sts -l '<label-selector>'

4. Read The Summary

Example:

expectedPrimary=suse-2 role=Standby workloads=1 nonZeroWorkloads=0 zeroWorkloads=1 desiredReplicas=0

Meaning:

• suse-2 is the management-plane expected primary cluster.
• The operator sampled 1 workload in suse-2.
• That workload has desired replicas set to 0.
• suse-2 is classified as Standby, which does not match the steady-state expectation for the primary cluster.

Repair Principles

Do not directly edit DisasterInstance.status.primaryCluster/secondaryCluster or the condition. First decide which cluster is the real service side, then align downstream replicas with the current expected relationship.

Common cases:

Case	Repair direction
BothStandby, expected primary should keep serving	Restore Deployments/StatefulSets in the expected primary cluster to their original replicas and keep the expected secondary at 0.
RoleReversed, business is running on expected secondary	First confirm whether a failover/reprotect just failed. Avoid accidental traffic movement. Then either align replicas back to the current expected direction or follow a controlled manual recovery process.
BothActiveObserved	This does not fail the instance, but confirm whether it came from an explicit source-scale-down skip. If dual-active was not intended, converge traffic and replicas.
Unknown	Fix cluster connectivity, permissions, namespaces, or selectors before interpreting role drift.

When real replicas return to:

expectedPrimary = Active
expectedSecondary = Standby

the operator sets RoleDrift=False on a later reconciliation. If the instance was Failed because of RoleDriftDetected, it recovers automatically:

• It returns to Protected when the current primary/secondary direction matches the basic configuration direction.
• It returns to Active when the current direction is the failover direction.

Prevention

• Do not manually scale the same Deployments/StatefulSets in downstream clusters while Failover, Reprotect, or Undo is running.
• If manual scaling is unavoidable, record the current primaryCluster/secondaryCluster first and ensure only the current primary keeps non-zero replicas at the end.
• Use separate drill namespaces so drill resources do not modify production standby namespaces.
• For production failover, keep external traffic-control records so replicas and traffic direction do not drift separately.
• When RoleDriftDetected appears, keep events and operation records until diagnosis is complete.