Discovery: Clients Cannot Find Services#

Symptom#

Clients fail before they reach any adapter: they cannot locate the services at all, or they locate some and not others. A client may work for one capability and fail for another, because it only found part of the deployment.

Blast radius#

Wide but shallow — the adapters themselves are usually healthy and reachable if addressed directly. Nothing is broken except the ability to find them, so a targeted test against a service’s own address is the fastest way to prove the adapters are fine.

Triage#

1. Ask discovery what it advertises.

For the deployment-wide service list, query the Discovery Service ingress hostname. Set TOKEN to your access token when authentication is enabled.

curl --fail -sS -H "Authorization: Bearer $TOKEN" https://<discovery-host>/api/v1/services | jq .

Compare the advertised service endpoints with what the deployment actually runs. A deployed service missing from the response indicates a registration or configuration problem.

To inspect API groups implemented by a specific Storage Service instance, query /v2alpha/services on that instance’s REST endpoint (port 8012 inside the cluster). This adapter capability response is separate from the deployment-wide Discovery Service list.

2. Empty or partial response.

Work outward from the service:

kubectl -n <namespace> get pods,svc,endpoints
kubectl -n <namespace> logs deployment/<discovery-deployment> --since=15m | tail -50

An endpoint list with no addresses means the Service selector matches no ready pods — the adapter is not ready, not missing.

3. Advertised but unreachable.

Discovery advertises what it is configured to advertise; it does not prove reachability. Test one advertised address directly:

curl -sS -o /dev/null -w '%{http_code}\n' https://<advertised-host>/<advertised-path>

If discovery lists it and the direct call fails, this is not a discovery problem — go to Loss of Connectivity: Backend and Egress or Certificate Expiry and TLS.

4. Stale after a change.

Clients cache discovery results. After a deployment change, a client may hold the previous view for the lifetime of its cache. Confirm by making the same query from a fresh client.

Fix#

  • Adapter not ready — resolve the adapter’s own failure; discovery follows once it is ready. First Response: Triage routes to the right runbook.

  • Misconfigured advertisement — correct the advertised host or path in your deployment values. Where a service group is version-gated, confirm the version you expect is actually enabled in that deployment.

  • Client caching — restart or refresh the client. Do not restart the server to work around a client-side cache.

Verify#

  • The Discovery Service’s /api/v1/services response lists the expected service endpoints.

  • Each Storage Service instance’s /v2alpha/services response lists its expected API groups.

  • A direct call to each advertised address succeeds.

  • A freshly started client reaches every capability it needs.