Known Issues and Troubleshooting

This page covers known product limitations and common deployment problems. For detailed per-phase troubleshooting steps, see the troubleshooting sections embedded in each deployment guide.

—

Known Issues

1.0.0

Data persistence: FILESERVICE_STATIC_DIR silently ignored

In Docker and Kubernetes deployments, the FILESERVICE_STATIC_DIR environment variable is injected but not consumed by the service entrypoint. Files are written to ephemeral /tmp/ storage and are lost on container or pod restart, even when a PVC is mounted.

Workaround: Pass the filesystem backend subcommand explicitly in your entrypoint so the service reads the env var:

python -m local_filesystem_service filesystem --static-dir /data/storage

IRSA web identity tokens not consumed on EKS

The Helm chart injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE but the Storage Service does not consume them. Deployments that rely on IRSA for pod-level S3 access will find the service falls back to the instance metadata credentials or fails if no other credentials are present.

Workaround: Inject credentials explicitly via a Kubernetes secret:

kubectl create secret generic storage-aws-credentials --from-literal=AWS_ACCESS_KEY_ID=<key> --from-literal=AWS_SECRET_ACCESS_KEY=<secret> -n storage-apis

Reference the secret in your Helm values under extraEnvs. See Storage Service Configuration for the full credentials configuration.

Notification events missed for writes outside the Storage API

If data is written directly to the underlying storage (S3, filesystem, etc.) without going through the Storage API, no update events are emitted. Clients and the storage service will not reflect those changes in a predictable timeframe.

Workaround: Use the Storage APIs for both read and write operations to ensure event consistency.

Kit streaming: authentication token expiry

The authentication token may expire and disconnect a Kit streaming session even though the streaming app is still active. This is planned for a future released Kit extension and Portal release.

Workaround: Start a new streaming session.

Kit / Client Library: Copy, Move, Rename, Create Folder not implemented

These operations are not yet available via the Storage API client library or Kit integration. Planned for a future release.

Workaround: Use Storage Navigator to perform these operations.

Multi-replica stale reads with caching enabled

When multiple Storage Service replicas run with caches enabled, clients may see stale reads for non-version-specific objects until the cache TTL expires. Use versioned access patterns to avoid this.

Azure Blob storage key location changed

In the early access releases, storage keys for the Azure Blob endpoints were configured with the /config/storage/azure/endpoints/<name>/storageKey property. As of version 1.0.0, this property is called /config/storage/azure/endpoints/<name>/credentials/storageKey/storageKey.

—

Common Deployment Problems

Ports already in use

Symptom: curl returns “Connection refused” even though the service appears to start, or a process fails to bind to port 8011, 50051, or 8080.

Cause: MicroK8s kube-proxy iptables rules may intercept traffic on those ports even when no explicit service is running, or a port-forward or Docker container from a previous phase is still active.

Fix:

# Check what is holding the port
ss -tlnp | grep -E '8011|50051|8080'

# Stop any remaining Docker container from the previous phase
docker stop localfilesystem-test

# Kill any lingering port-forward sessions
pkill -f "kubectl port-forward" || true

MicroK8s TLS certificate mismatch

Symptom: kubectl port-forward fails with an x509 error or references mismatched IPs. kubectl logs and kubectl exec also fail.

Cause: The kubelet TLS certificate was issued before a VPN interface (for example, GlobalProtect) was added. The new interface IP is not in the certificate’s Subject Alternative Names.

Fix:

# Verify the kubelet cert SANs
openssl x509 -in /var/snap/microk8s/current/certs/kubelet.crt \
    -noout -text | grep -A 1 "Subject Alternative"

# If your current IP is missing, regenerate:
sudo microk8s refresh-certs --cert kubelet.crt
sudo microk8s stop && sudo microk8s start

API server certificate mismatch:

sudo microk8s refresh-certs --cert server.crt
sudo microk8s stop && sudo microk8s start

Broken percent-encoding in curl URLs

Symptom: 404 Not Found or 400 Bad Request when addressing a file object via curl.

Cause: The file-storage:// scheme must be double-percent-encoded. A common mistake is file-storage%3A%2F%fileservice (missing one 2).

Correct encoding:

# file-storage://fileservice/hello.txt encodes to:
file-storage%3A%2F%2Ffileservice%2Fhello.txt

# Full example:
curl http://localhost:8011/v1beta/fileobject/by-address/file-storage%3A%2F%2Ffileservice%2Fhello.txt

Discovery Service returns empty endpoints

Symptom: GET /api/v1/services returns an empty list or the storage endpoints are missing.

Fix (in order):

# 1. Check all pods are running
microk8s kubectl get pods -n storage-apis-dev

# 2. Verify services and endpoints are wired
microk8s kubectl get svc,endpoints -n storage-apis-dev

# 3. Confirm port-forward sessions are still active
ps -ef | grep "kubectl port-forward" | grep -v grep

# 4. Check Discovery Service logs for configuration errors
microk8s kubectl logs deployment/discovery-service -n storage-apis-dev

Port-forward session drops silently

Symptom: Requests that worked moments ago now return “Connection refused” without any error from the port-forward command.

Cause: kubectl port-forward sessions drop when the pod restarts or after idle timeout. They do not automatically reconnect.

Fix: Rerun the port-forward commands. For persistent local development, consider running them in a terminal multiplexer (tmux/screen) or using a loop:

while true; do
    microk8s kubectl port-forward -n storage-apis-dev service/storage-service 8011:8011
    sleep 2
done