Troubleshooting and Error Behavior#

User Info Service is a read-only service that answers directory lookups over REST and gRPC. This page describes the signals an operator uses to tell whether an instance is healthy and serving, the error responses a caller can receive and what each one means, why some requests can fail while others keep working depending on where the data is served from, and how to diagnose the problems these signals point to.

It builds on the roles and data flow in Directory Synchronization and Data Freshness, the telemetry in Observability, and the deployment steps in User Info Service Configuration. For the availability behavior behind an outage of one component, see Scaling and High Availability.

Health and Readiness Probes#

The service exposes two HTTP probes on the REST port, and a gRPC health service on the gRPC port, that together tell an operator whether an instance is running and whether it can serve directory data. Both roles serve all three; what “ready” means differs by role.

  • Liveness (GET /healthz). Returns a 2xx status whenever the process is running and able to answer HTTP. It does not reflect whether the directory is loaded, so a live process that has not yet become ready still passes liveness. A liveness failure means the process is wedged, and Kubernetes restarts the container.

  • Readiness (GET /readyz). Returns a 2xx status only when the instance can serve directory data, and a 503 Service Unavailable until then. For a reader, readiness turns on once it has loaded a directory from the shared store; for the writer, once its first full synchronization has completed. A reader that is not ready is removed from the Kubernetes Service and receives no traffic.

  • gRPC health. The gRPC port exposes the standard gRPC Health Checking Protocol. The service reports NOT_SERVING until the instance is ready and SERVING afterward, mirroring the readiness probe for gRPC clients and gRPC-aware load balancers.

The most common readiness symptom is the writer taking a long time to become ready on a large directory: it reports ready only after its first full synchronization finishes, which can take hours when membership synchronization is enabled. If the readiness failure budget is too small, Kubernetes restarts the writer before that first synchronization completes, and it never makes progress. Size the budget to the directory as described in Memory and CPU Sizing and under Configuration below. Readers, by contrast, become ready in seconds by loading whatever the store already holds; a reader that starts against an empty store becomes ready but serves an empty directory until the writer has saved data and the reader has reloaded it.

Error Responses#

Every non-2xx REST response carries a JSON body with a stable machine-readable error code and a short human-readable message, and the equivalent gRPC call returns the matching status code. The codes and their gRPC equivalents are:

REST status

REST error

gRPC status

Meaning

401

unauthorized

UNAUTHENTICATED

The bearer token is missing or invalid, or the caller’s identity could not be resolved from it.

403

forbidden

PERMISSION_DENIED

The caller is authenticated but not permitted to perform the operation.

404

not_found

NOT_FOUND

The requested user, group, or membership does not exist.

429

rate_limited

RESOURCE_EXHAUSTED

The identity provider’s rate limit was exceeded. REST responses include a Retry-After header.

502

bad_gateway

UNAVAILABLE

The identity provider returned an unexpected error.

500

internal_error

INTERNAL

An unexpected internal failure, or the Permission Service could not be reached to authorize the request.

The client-visible message is a fixed, generic string; it never echoes identity-provider diagnostics, internal error chains, or caller-supplied input back to the client. The detailed cause is recorded only in the service’s logs, so when a 5xx response needs investigating, the matching log line on the instance that produced it carries the specifics. Server-side errors (bad_gateway, internal_error) are logged at error level and a rate_limited response at warning level, so both surface on dashboards; the expected client-side statuses (unauthorized, forbidden, not_found) are logged at debug level and are not treated as faults. See Observability for the request metrics and logs that expose these responses in aggregate.

Token validation itself is performed by the platform’s authentication gateway in front of the service; a 401 from the service means the request reached it without a usable token or with one whose subject does not match a directory record, not that a specific token was rejected by internal validation.

Why Some Requests Fail While Others Keep Working#

Whether a request depends on the identity provider being reachable is decided entirely by where the requested data is served from, so a provider problem can affect some requests while leaving others untouched.

  • Data served from the replica. In the synchronized deployment, readers answer user, group, and membership lookups from their in-memory copy of the directory. These requests never contact the identity provider, so they keep succeeding — returning the last loaded data — even while the provider is unreachable, rate-limiting, or returning errors. A provider outage pauses data freshness rather than availability; see High Availability.

  • Data served live from the provider. When a data category is not cached — for example, when membership synchronization is disabled — requests for that kind of data are answered directly from the identity provider instead of from the replica. Those specific requests can therefore return 429 rate_limited or 502 bad_gateway when the provider throttles or fails, while user and group lookups served from the replica continue to succeed. The same is true of the reader-only live mode, in which every request goes to the provider and provider problems surface on all of them; see Live Mode.

  • Authorization dependency. When Permission Service integration is enabled, each request is authorized before it runs and list results are filtered per item. If the Permission Service cannot be reached, the service fails closed and returns 500 internal_error even though the directory data is available. Health, readiness, and current-user requests are exempt from permission checks and keep working. See Authorization with the Permission Service.

Diagnosing Common Problems#

The signals above localize most problems to a role and a cause:

  • No instance is ready. Check readiness across the fleet. If readers are unready, they have no directory to load — confirm the writer has completed a synchronization and the shared store holds data; see Database Configuration. If the writer is unready long after startup, its first full synchronization has not finished — check the writer’s synchronization logs and metrics and confirm the readiness budget covers the initial synchronization.

  • Requests return 429 or 502. These come from the identity provider and appear only on requests served live from it — live mode, or a data category that is not cached. Check the Microsoft Graph client metrics and the writer’s synchronization logs for throttling or provider errors. Data served from the replica is unaffected. See Microsoft Entra ID Integration.

  • Requests return 500 with authorization enabled. A fail-closed internal_error most often means the Permission Service is unreachable or misconfigured. Confirm its address and that it holds policies for the configured service name; see Permission Service Integration.

  • Data looks stale or differs between callers. This is expected for the service’s eventually-consistent model: a change becomes visible only after the writer’s next cycle saves it and each reader’s next reload picks it up, and readers reload independently. See Data Freshness and Eventual Consistency.

In every case, the per-instance logs carry the detailed cause behind a generic client message, and the request and Microsoft Graph client metrics show whether errors are concentrated on a particular route, role, or upstream call.

Configuration#

The probe paths are fixed at /healthz and /readyz on the REST port; the probe timing is configurable through the Helm chart and applies to both the reader and the writer. The readiness failure budget is the setting most often adjusted for troubleshooting, because it governs how long the writer is allowed to complete its first full synchronization before Kubernetes restarts it.

Setting

Helm value

Default

Description

Liveness probe

livenessProbe

httpGet on /healthz

HTTP liveness probe run against the REST port.

Readiness probe

readinessProbe

httpGet on /readyz

HTTP readiness probe run against the REST port.

Readiness initial delay

readinessProbe.initialDelaySeconds

5

Seconds to wait after container start before the first readiness check.

Readiness period

readinessProbe.periodSeconds

10

Interval between readiness checks.

Readiness failure budget

readinessProbe.failureThreshold

180

Consecutive failed readiness checks tolerated before Kubernetes restarts the container. Raise to cover a long initial synchronization.

The default budget tolerates roughly initialDelaySeconds + failureThreshold × periodSeconds (about 30 minutes) before a restart. Raise failureThreshold when the writer’s initial full synchronization can take longer than that — which membership synchronization on a large directory can — as covered in Memory and CPU Sizing.

Configuring Probes in the Helm Chart#

Add the probe settings to userinfo-values.yaml. The example below keeps the default liveness probe and widens the readiness budget so the writer can finish a long initial synchronization on a large directory:

livenessProbe:
  httpGet:
    path: /healthz
    port: http

readinessProbe:
  httpGet:
    path: /readyz
    port: http
  initialDelaySeconds: 5
  periodSeconds: 60
  failureThreshold: 300   # 5 + 300×60 ≈ 5 hours for a long initial sync

Install or upgrade the release with the same commands used for the synchronized directory deployment in User Info Service Configuration.

References#