Velero BSL Unavailable: Pod Network Forwarding Troubleshooting

This runbook records a case where DisasterConfig entered Error because the target cluster Velero BackupStorageLocation was Unavailable.

It applies when:

• The management cluster BSL is Available.
• The target cluster BSL is Unavailable.
• The object storage endpoint is reachable from the node host, but times out from normal Pods.
• The Velero Pod is running on an affected node.

Symptom

DisasterConfig status is Error, and operator logs contain:

BackupStorageLocation <storage-repository>-<source-cluster> is in Unavailable status

The target cluster BSL status message contains an S3 timeout:

BackupStorageLocation "<bsl-name>" is unavailable:
operation error S3: ListObjectsV2 ...
Get "http://<s3-endpoint>/<bucket>?...": dial tcp <s3-endpoint>: i/o timeout

Key Point

Do not assume the management cluster BSL is the failing one.

The operator applies StorageRepository to the target cluster and waits for the target Velero BSL to become Available. Therefore:

1. The management cluster BSL can be healthy.
2. The error may point to the target cluster.
3. The target cluster Velero Pod must be able to reach the object storage endpoint from Pod networking.

1. Confirm The DisasterConfig Target

KUBECONFIG=<management-kubeconfig>
DC=<disasterconfig-name>

kubectl get disasterconfig.testudo.softcdata.com "$DC" \
  -o jsonpath='sourceCluster={.spec.sourceCluster}{"\n"}targetCluster={.spec.targetCluster}{"\n"}storageRepository={.spec.storageRepository}{"\n"}status={.status.status}{"\n"}message={.status.message}{"\n"}'

Check:

• spec.sourceCluster
• spec.targetCluster
• spec.storageRepository
• status.message

2. Compare BSL Status

Management cluster:

kubectl get backupstoragelocations.velero.io -A -o wide

Target cluster:

kubectl --kubeconfig=<target-kubeconfig> \
  get backupstoragelocations.velero.io -A -o wide

If the management BSL is Available but the target BSL is Unavailable, troubleshoot the target cluster.

3. Inspect The Target BSL Error

kubectl --kubeconfig=<target-kubeconfig> \
  get backupstoragelocation.velero.io -n velero <bsl-name> \
  -o jsonpath='phase={.status.phase}{"\n"}message={.status.message}{"\n"}lastValidation={.status.lastValidationTime}{"\n"}'

If the message contains:

dial tcp <s3-endpoint>: i/o timeout

Velero cannot reach object storage from its runtime environment.

4. Find The Velero Pod Node

kubectl --kubeconfig=<target-kubeconfig> \
  get pod -n velero -o wide

Record the node:

velero-xxxxx   Running   ...   NODE=<affected-node>

5. Reproduce From A Normal Pod

Schedule a temporary Pod on the same node:

K=<target-kubeconfig>
NODE=<affected-node>
S3_ENDPOINT=<s3-endpoint-host>
S3_PORT=<s3-endpoint-port>
POD=netcheck-${NODE}

kubectl --kubeconfig=$K delete pod -n default $POD --ignore-not-found

kubectl --kubeconfig=$K run $POD -n default \
  --image=nginx:alpine \
  --restart=Never \
  --overrides="{\"spec\":{\"nodeName\":\"$NODE\"}}" \
  --command -- sh -c "nc -vz -w 5 $S3_ENDPOINT $S3_PORT 2>&1 || true; wget -S -T 8 -O - http://$S3_ENDPOINT:$S3_PORT/ 2>&1 | head -n 20 || true"

kubectl --kubeconfig=$K logs -n default $POD --tail=80
kubectl --kubeconfig=$K delete pod -n default $POD --ignore-not-found

Timeout usually looks like:

nc: <s3-endpoint> (<s3-endpoint>:<port>): Operation timed out
wget: download timed out

If the root endpoint returns 403 Forbidden, that is usually a good sign: the network reached object storage, but the request was not S3-signed.

6. Compare hostNetwork

If node-level ping or telnet works but normal Pod access times out, compare with a hostNetwork Pod:

POD=netcheck-${NODE}-host

kubectl --kubeconfig=$K delete pod -n default $POD --ignore-not-found

kubectl --kubeconfig=$K run $POD -n default \
  --image=nginx:alpine \
  --restart=Never \
  --overrides="{\"spec\":{\"nodeName\":\"$NODE\",\"hostNetwork\":true,\"dnsPolicy\":\"ClusterFirstWithHostNet\"}}" \
  --command -- sh -c "nc -vz -w 5 $S3_ENDPOINT $S3_PORT 2>&1 || true; wget -S -T 8 -O - http://$S3_ENDPOINT:$S3_PORT/ 2>&1 | head -n 20 || true"

kubectl --kubeconfig=$K logs -n default $POD --tail=80
kubectl --kubeconfig=$K delete pod -n default $POD --ignore-not-found

Interpretation:

Normal Pod	hostNetwork Pod	Meaning
Timeout	Works	Pod network forwarding, NAT, CNI, or node firewall problem
Timeout	Timeout	Node-to-storage network or object storage route problem
Works	Works	BSL issue may be credentials, bucket, region, CA, or path-style config

7. Move Velero Temporarily

If only one node has broken Pod networking, temporarily move Velero to a healthy node or cordon the affected node and restart Velero:

kubectl --kubeconfig=<target-kubeconfig> cordon <affected-node>
kubectl --kubeconfig=<target-kubeconfig> -n velero delete pod -l deploy=velero
kubectl --kubeconfig=<target-kubeconfig> -n velero get pod -o wide

Then re-check BSL:

kubectl --kubeconfig=<target-kubeconfig> -n velero get backupstoragelocation <bsl-name> -o wide

If BSL becomes Available, the object storage configuration is likely correct and the issue is local to Pod networking on the original node.

8. Permanent Fixes

Check the target cluster networking stack:

• CNI health and kube-proxy rules.
• Node firewall and forwarding settings.
• SNAT/masquerade rules from Pod CIDR to the object storage network.
• NetworkPolicy or egress firewall rules.
• Routes from Pod CIDR to the S3 endpoint.
• MTU mismatch if large transfers hang.

After fixing networking, restart Velero or wait for BSL validation:

kubectl --kubeconfig=<target-kubeconfig> -n velero rollout restart deploy/velero
kubectl --kubeconfig=<target-kubeconfig> -n velero get backupstoragelocation <bsl-name> -w

Conclusion

When BSL is unavailable only on the target cluster and the error is i/o timeout, validate from the same network position as Velero: a normal Pod on the same node. Host-level connectivity is not enough to prove Velero can access object storage.