Operator Runbooks#

These runbooks are for operators running a deployed Storage APIs stack. Each one starts from a symptom you can observe — an alert, a failing request, a user report — and walks to a fix.

They are deliberately different from Known Issues and Troubleshooting, which lists product limitations and first-deployment problems. A runbook assumes the deployment worked and something has since changed: a credential rotated, a certificate expired, a database became unreachable, a queue backed up.

How to use these#

Every runbook follows the same shape, so you can skim to the part you need:

Symptom: What you see. Match on this first.
Blast radius: What is affected while the problem persists, and what is not. Read this before acting — several of these failures degrade one capability while leaving the rest of the stack healthy, and an unnecessary restart makes things worse.
Triage: Read-only commands that confirm the cause. None of these change state.
Fix: The change that resolves it, and whether it belongs in a deployment change or is safe to apply directly.
Verify: How to confirm the fix worked, from the outside.

Important

Triage steps are safe to run at any time. Fix steps are not. Where a fix mutates a live deployment, the runbook says so explicitly and gives the declarative equivalent, because a change made by hand is reverted by the next deploy.

Before you start#

Two facts make most of these faster:

Identify the failing tier. The stack is clients → service adapters → infrastructure. A failure in the infrastructure tier (database, queue, object store, identity provider) usually presents as an adapter error, so the adapter logs name the real cause. Read those first.
Check whether it is one capability or the whole service. Storage Service, for example, keeps serving reads when its event subscription is broken, and keeps serving unversioned operations when its metadata database is down. Knowing which capability failed narrows it to one runbook immediately.

Service reference#

Ports and health endpoints referenced throughout these runbooks.

Service	gRPC	REST/HTTP	Health
Storage Service	8011	8012	gRPC health on 8100
Permission Service	3010	3000	see service docs
Event Aggregation Service	50051	8001	see service docs
Event Consumer Service	—	see chart	see service docs
Discovery Service	—	8080 (`/api/v1/services`)	see chart

Storage Service also exposes /v2alpha/services on its REST port 8012 to describe that adapter’s API groups. This is separate from the deployment-wide Discovery Service endpoint.