Secret Rotation and Credential Expiry#

Symptom#

Something that worked yesterday now fails with an authentication or authorization error, and no deployment changed. Typical shapes:

  • Storage backend calls fail with an access-denied error from S3 or Azure Blob.

  • A service cannot authenticate to its database.

  • Service-to-service calls fail with an invalid or expired token.

  • A queue consumer stops receiving, with an authorization error in its logs.

Blast radius#

Scoped to whatever used the rotated credential — but that can be wide. A rotated backend credential takes out all access to that backend while leaving every other endpoint working, which presents as “some content is unavailable” rather than an outage.

Credentials in this stack come from several places, and they fail differently:

Credential

Where it comes from

Failure shape

Object-store keys

Deployment values or a Kubernetes secret

Backend access denied

Workload identity (AKS / EKS IRSA)

Projected service-account token

Token expiry, or trust misconfigured

Service identity token

Access-token file provided by a sidecar

Calls rejected as unauthenticated

OAuth2 client secret

Deployment values or secret

Token endpoint rejects the client

Database credentials

ExternalSecret from a vault or key vault

Authentication failure at connect

Queue credentials

Access-token file or connection secret

Consumer stops receiving

Triage#

1. Confirm it is credentials, not permissions.

An expired credential and a revoked grant look similar from the client. The service logs distinguish them: authentication failures name the credential, authorization failures name the action or resource.

2. Is the secret present and current in the cluster?

kubectl -n <namespace> get externalsecret
kubectl -n <namespace> describe externalsecret <name> | tail -20
kubectl -n <namespace> get secret <name> -o jsonpath='{.metadata.creationTimestamp}{"\n"}'

An ExternalSecret whose status is not SecretSynced is the whole answer — the vault side failed, and the cluster is still serving the last value it synced, which is now stale.

3. Has the pod picked it up?

This is the step most often missed. A secret can be correct in the cluster and stale in the process:

  • A secret mounted as a file is updated in place, but only if the consuming code re-reads it.

  • A secret injected as an environment variable is fixed for the life of the pod. Rotating it requires a restart.

# When did the pod start, relative to when the secret changed?
kubectl -n <namespace> get pods -o custom-columns='NAME:.metadata.name,START:.status.startTime'

If the pod started before the secret changed and the value is injected as an environment variable, the pod is running the old credential regardless of what the secret now contains.

4. Is the token simply expired rather than rotated?

Short-lived tokens — workload identity, access-token files — expire on a schedule. If the failure recurs at a regular interval, the refresh mechanism is broken rather than the credential.

kubectl -n <namespace> exec deployment/<service> -- \
    sh -c 'ls -l --time-style=full-iso <token-path>'

A token file whose modification time has stopped advancing means the provisioning sidecar has stopped refreshing it.

Fix#

  • ExternalSecret not syncing — fix the vault-side cause (missing key, expired vault credential, changed path), then let it resync. Deleting the Kubernetes secret to force a resync is safe only once the source is fixed; deleting it while the source is broken removes the last working value.

  • Pod holding a stale value — roll the deployment. Prefer a rollout restart over deleting pods, so surge and readiness are respected:

    kubectl -n <namespace> rollout restart deployment/<service>
kubectl -n <namespace> rollout status deployment/<service>

  • Rotated backend key — update it through your deployment values so the change survives the next deploy, not by editing the live secret.

  • Expired workload identity — check the federated credential and the service-account annotation still match the identity; an identity replaced on the cloud side leaves both looking present but no longer trusted.

Warning

Do not patch credentials directly into a running deployment as a permanent fix. The next deploy reconciles from source and the failure returns, typically at the least convenient moment and with no record of what changed.

Verify#

  • The failing operation succeeds again.

  • ExternalSecrets report SecretSynced.

  • The failure does not recur at the next rotation interval — which is the real test for expiry-driven cases, and worth scheduling a check for rather than assuming.