Loss of Connectivity: Backend and Egress#

Symptom#

A service cannot reach something outside itself: the object store, a queue, an identity provider, or the internet generally. Requests hang and then time out, or fail with connection-refused, DNS resolution failures, or TLS handshake timeouts.

Blast radius#

Depends on what became unreachable. The important distinction is hang versus fail: a hard refusal fails fast and degrades one capability, while a network black hole causes requests to occupy connections until they time out, which can exhaust capacity and make the service look entirely down.

Triage#

1. Resolve, then connect, then authenticate — in that order.

Most of the time this identifies the layer immediately.

# DNS
kubectl -n <namespace> exec deployment/<service> -- nslookup <host>

# TCP
kubectl -n <namespace> exec deployment/<service> -- sh -c 'nc -zv <host> 443'

# TLS + HTTP
kubectl -n <namespace> exec deployment/<service> -- \
    sh -c 'curl -sS -o /dev/null -w "%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s\n" https://<host>/'

2. Is it this pod, this namespace, or the cluster?

Run the same check from a throwaway pod in the same namespace, then in another namespace. A failure that follows the namespace points at a NetworkPolicy; one that follows the node points at the node’s routing or egress.

kubectl -n <namespace> get networkpolicy
kubectl -n <namespace> describe networkpolicy <name>

3. Is egress filtered?

Clusters with restricted egress allow a specific set of destinations. A newly added backend, region, or identity provider endpoint fails while everything previously configured keeps working — the tell is that the failure is specific to one host, and began when configuration changed rather than spontaneously.

4. Is it saturation rather than reachability?

If the host is reachable but slow, and errors are timeouts rather than refusals, look at throughput and connection counts before blaming the network. A backend that is rate-limiting will often return explicit throttling responses — check the adapter logs for them.

Fix#

  • DNS — correct the name or the cluster DNS configuration. A recently changed service or endpoint name is the usual cause.

  • Blocked egress — add the destination to the allowed set through your infrastructure configuration, not by disabling the policy.

  • Backend outage — confirm with the provider’s status before changing anything. The adapters recover on their own when the backend returns; restarting them during an upstream outage adds cold caches to an existing problem.

  • Throttling — reduce concurrency at the source. Request throttling and egress limits are configurable on Storage Service; raising client parallelism against a throttling backend makes throughput worse, not better.

Note

Reads that go through a presigned-URL redirect are fetched by the client, not the service. If service-side checks pass but clients still fail, test egress from where the client actually runs — the two paths do not share a route.

Verify#

  • DNS, TCP and TLS checks above all succeed from inside the affected pod.

  • The capability that depended on the destination works again.

  • Error rate and latency return to their normal range, rather than merely moving off the alert threshold.