Database Connectivity Loss

Symptom

A service that depends on a database starts failing while the rest of the stack stays healthy:

• Storage Service — reads and writes succeed, but user metadata operations, version history or optimistic-locking operations fail.

• Permission Service — every authorization decision fails, so callers see permission denials across all services that consult it.

• User Info Service — user and group lookups fail or return stale results.

Blast radius

Depends on which database:

Database	What fails
Storage metadata (PostgreSQL / CosmosDB)	User metadata and version-aware operations. Plain object read and write continue.
Permission Service	Authorization decisions. Because other services call it, this presents as a stack-wide outage even though only one component is down.
User Info Service	Identity lookups. Anything resolving a user or group by name degrades.

The Permission Service case is the one to recognise quickly: a database failure in one component looks like a global permissions outage.

Triage

1. Confirm it is the database, not the service.

kubectl -n <namespace> logs deployment/<service> --since=15m | grep -i -E 'connection|timeout|refused|pool|postgres|cosmos'

Connection-pool exhaustion, refused connections and authentication failures all name themselves here, and they mean different things — the next steps differ.

2. Is the database itself up?

For a CloudNativePG-managed cluster:

kubectl -n <pg-namespace> get cluster
kubectl -n <pg-namespace> get pods -l cnpg.io/cluster=<cluster-name>
kubectl -n <pg-namespace> describe cluster <cluster-name> | tail -30

A cluster stuck without a primary, or a pod that cannot attach its volume, explains the outage directly.

3. Is it reachable from the service?

kubectl -n <namespace> exec deployment/<service> -- \
    sh -c 'nc -zv <db-host> 5432' 2>&1 | tail -2

Reachability failing while the database is healthy points at network policy, DNS, or a changed service name — see Loss of Connectivity: Backend and Egress.

4. Did the credential change?

An authentication failure that begins abruptly, with no deployment change, is almost always a rotated secret that the service has not picked up. See Secret Rotation and Credential Expiry.

kubectl -n <pg-namespace> get externalsecret
kubectl -n <pg-namespace> get secret <db-secret> -o jsonpath='{.metadata.creationTimestamp}'

5. Is it capacity rather than availability?

Connection-pool exhaustion under load looks like an outage but is a sizing problem. Check connection counts against the server’s limit before restarting anything:

kubectl -n <pg-namespace> exec <pg-primary-pod> -c postgres -- \
    psql -U postgres -tAc "select count(*), (select setting from pg_settings where name='max_connections') from pg_stat_activity;"

Fix

• Database down — recover the database cluster first; the adapters reconnect on their own once it is reachable. Do not restart the adapters while the database is down; they will crash-loop and tell you less.

• Credentials rotated — follow Secret Rotation and Credential Expiry. Restarting the consuming pod is usually required only if it caches the credential at startup.

• Pool exhausted — raise the pool size or the server’s connection limit in your deployment values, and look for the caller that changed behaviour. A sudden pool exhaustion with unchanged configuration is a workload change, not a configuration fault.

• Network path broken — see Loss of Connectivity: Backend and Egress.

Warning

Deleting and recreating a database cluster to “reset” it destroys its volume and its data. For the metadata and permissions databases that is a data-loss event, not a restart. Recover the existing cluster instead, and take a backup before any destructive step.

Verify

• The service’s own logs show successful connections and stop reporting pool or timeout errors.

• The specific capability returns: user metadata reads succeed, authorization decisions resolve, or identity lookups answer.

• Connection count sits well below the server limit under normal load.