Certificate Expiry and TLS#

Symptom#

Requests that previously worked now fail with a TLS error, and the error names a certificate: expired, not yet valid, hostname mismatch, or unknown authority. Depending on where the bad certificate sits, this is either external-only or stack-wide.

Blast radius#

Certificate

What fails

Ingress / HTTPProxy serving certificate

External clients that connect through the affected ingress endpoint. Requests that bypass ingress may still succeed.

Identity provider (OIDC) signing certificate

Token validation across every service that authenticates callers.

Object-store or database server certificate

The adapter that talks to it, only.

Client certificate presented by a service

That service’s outbound calls.

Kubernetes API server certificate

kubectl and in-cluster controllers. Usually a development-cluster problem.

Triage#

1. Read the certificate actually being served.

echo | openssl s_client -connect <host>:443 -servername <host> 2>/dev/null \
    | openssl x509 -noout -subject -issuer -dates

notAfter in the past confirms expiry. A subject/CN or SAN list that does not include the hostname being used is a mismatch, not an expiry — a different fix.

2. Confirm the boundary.

If an internal request succeeds while external access fails, investigate the external access path. This comparison alone does not identify a certificate problem.

Check the original client’s error for certificate expiry, hostname mismatch, or an untrusted issuer or chain. Confirm the certificate served by the failing hostname. For DNS errors, connection failures, or HTTP errors, investigate the corresponding network, ingress, or authentication configuration.

The following probe checks internal HTTP reachability; it does not validate external TLS:

kubectl -n <namespace> run curl-probe --rm -it --restart=Never --image=curlimages/curl -- \
    curl -sS -o /dev/null -w '%{http_code}\n' http://storage-service:8012/v2alpha/services

3. Check what issued it and whether renewal is running.

For cert-manager-managed certificates:

kubectl -n <namespace> get certificate
kubectl -n <namespace> describe certificate <name> | tail -30
kubectl -n <namespace> get certificaterequest,order,challenge

A Certificate that is Ready=False with a failing challenge tells you renewal has been failing for a while — expiry is the symptom, the failed challenge is the cause.

4. Identity provider certificates.

Token validation failures after an IdP key rollover are not a deployment fault. Confirm the provider’s JWKS is reachable and current from inside the cluster; a service that cached the old key set and cannot refresh it will reject every token until restarted.

Fix#

  • Renewal broken — fix the issuer or the challenge path, then let renewal complete. Forcing a new certificate while the challenge still fails produces the same expired state again.

  • Genuinely expired with working renewal — trigger renewal and confirm the new certificate is served. With cert-manager, deleting the Certificate’s secret causes reissue; do this only when you have confirmed issuance works, because until reissue completes nothing is served.

  • Hostname mismatch — correct the SAN list or the hostname clients use. Reissuing the same certificate does not fix a mismatch.

  • IdP rollover — restart the services that cache the key set, after confirming the new JWKS is reachable.

Warning

Disabling TLS verification to “unblock” a failure removes the protection the certificate exists to provide, and tends to outlive the incident. If you need it to prove a diagnosis, remove it in the same session.

Verify#

echo | openssl s_client -connect <host>:443 -servername <host> 2>/dev/null \
    | openssl x509 -noout -dates

  • notAfter is in the future, with enough margin that renewal will run well before it.

  • The previously failing client succeeds without any verification override.

  • kubectl get certificate reports Ready=True.