Read Path: Failures and Slow Reads#

Symptom#

Reads through the Storage API fail, or succeed but are much slower than usual. Writes may or may not be affected — if writes fail too, the cause is more likely backend-wide, so start with Loss of Connectivity: Backend and Egress.

Blast radius#

Reads are the highest-volume operation in the stack, so a read-path failure is usually the first thing users notice. It is also the path with the most moving parts: request routing, permission evaluation, cache lookup, backend fetch, and — for larger objects — a presigned redirect the client follows directly to the object store.

That last point matters: a read can fail entirely outside the service, at the client’s own network path to the backend.

Triage#

1. Establish which stage fails.

# Stat only — no data transfer, exercises routing, permissions and metadata
curl -sS -o /dev/null -w '%{http_code}\n' -X HEAD \
    -H "Authorization: Bearer $TOKEN" \
    "https://<host>/v1alpha/fileobject/by-address/<url-encoded-address>"

If stat succeeds and the read fails, the failure is in data transfer, not in addressing or permissions.

2. Read the service logs for the backend’s own error.

kubectl -n <namespace> logs deployment/storage-service --since=15m | grep -i -E 'error|denied|timeout|throttl'

Adapters surface the backend’s error rather than masking it. Access-denied points at Secret Rotation and Credential Expiry; timeouts at Loss of Connectivity: Backend and Egress; throttling at the backend’s own limits.

3. Distinguish slow from failing.

Latency that rises with object size points at transfer, not at the service. Latency that is uniform across sizes points at a per-request cost — permission evaluation, metadata lookup, or a cache that is missing.

4. Check the cache is doing its job.

A sudden latency increase with no other change often means the cache is cold or not being populated. A cache that was invalidated by a flood of events, or one whose backing Redis is unreachable, degrades to origin reads for everything.

kubectl -n <namespace> logs deployment/storage-service --since=15m | grep -i -E 'cache|redis'

5. Redirect reads.

For sizes served by presigned redirect, the service returns a URL and the client fetches it. If the service’s own checks pass but clients still fail, test the redirect target from the client’s network — see the note in Loss of Connectivity: Backend and Egress.

Fix#

Backend credentials — Secret Rotation and Credential Expiry.
Backend unreachable or throttling — Loss of Connectivity: Backend and Egress. Reduce concurrency rather than increasing it.
Cache tier down — restore Redis; reads continue from origin meanwhile, slower. Do not disable caching as a permanent response to a cache outage.
Stale content rather than failure — the read path is working and the invalidation path is not. Go to Notifications: Delayed, Missing or Duplicated.

Verify#

stat and full read both succeed for a known object.
Latency returns to its normal distribution, checked across small and large objects.
Cache hit rate recovers, rather than merely error rate falling.