Live Edit operations#

Use this guide after deploying Live Edit. It covers operational settings and checks for the current release; the shipped chart’s values.yaml lists the complete configuration surface.

Scaling and limits#

The chart enables autoscaling from 3 to 10 replicas and a disruption budget with minAvailable: 2. Its CPU and memory utilization targets are both 80% of resource requests, not limits. Tune pod resources and the autoscaler together using measured workloads. For a fixed replica count, disable autoscaling and set replicaCount; keep the disruption budget consistent with that count.

Helm setting	Default	Operational effect
`valkey.subscriber_pool_size`	256	Active layer subscriptions per replica; additional subscriptions wait for capacity.
`valkey.pool_size`	64	Pool size for backing-store operations; affects connection demand.
`layer.max_size`	104857600 bytes	Maximum layer-upload size.
`shutdown.timeout_secs`	5 seconds	Initial shutdown drain allowance.
`shutdown.overall_budget_secs`	30 seconds	Subsequent cleanup budget.
`terminationGracePeriodSeconds`	40 seconds	Kubernetes pod termination allowance.

New replicas can receive new requests without session affinity, but adding replicas does not move existing long-lived streams. Clients must reconnect after pod termination, backend interruptions or ingress timeouts. Keep the pod termination allowance longer than the two shutdown budgets combined when changing them.

Monitoring#

Prometheus metrics are served at /metrics on port 9090 independently of OTLP export. To enable a ServiceMonitor, install the Prometheus Operator CRDs first and set serviceMonitor.enabled: true with the labels required by your monitoring stack. For OTLP/gRPC export, configure each signal independently:

observability:
  metrics:
    enabled: true
    endpoint:
      grpc: "http://otel-collector.observability.svc.cluster.local:4317"
  tracing:
    enabled: true
    endpoint:
      grpc: "http://otel-collector.observability.svc.cluster.local:4317"

Each enabled signal needs its endpoint.grpc; the chart does not use endpoint.http for these exports. Monitor readiness and RPC failures together with Valkey memory, latency and connection usage. For checkpoint deployments, watch live_edit_checkpoint_writes_total{result="error"}, live_edit_checkpoint_oldest_pending_age_seconds and live_edit_service_token_available. A healthy pod or successful Ping does not prove that a user can access a base asset or that checkpoints can be written.

Checkpoints and recovery#

Use a shared StorageApi backup location for a multi-replica deployment. LocalDisk is intended for deployments that deliberately provide suitable persistent storage; a container filesystem or independent per-pod disks are not a shared production backup store. Never use the InMemory adapter for persistent recovery.

Backups run asynchronously. The default checkpoint interval is 60 seconds, but that is a scheduling setting rather than a maximum data-loss guarantee. Busy layers, backing-store latency, failed uploads and token problems can delay completed backups. Edits newer than the latest recoverable backup can be lost after Valkey loss.

Before relying on recovery:

Edit a disposable live layer and verify successful checkpoint writes and objects under the configured backup prefix.
Check that the checkpoint service identity can read, list, write and delete there, including transfers to any redirected object-storage endpoints.
Test recovery in an isolated deployment using a copy of the backups and an empty backing store. Verify sessions and layer content with both fresh and reconnecting clients, not only pod readiness.
Measure the recovery time and recovered content against your requirements.

Warning

After recovery, reconnecting clients can retain stale content, including prims created after the recovered backup. Before resuming editing, reload the recovered live layer from empty client state rather than resuming its cached state. Keep layer.catchup_snapshot_enabled: true; disabling it can also leave reconnecting clients with pre-recovery content. Verify the reload procedure with your client during recovery testing.

Do not remove individual backup objects or apply bucket lifecycle rules without accounting for retained checkpoint generations and their dependent content. Keep one consistent backup prefix and configuration across the replicas of a deployment. Do not point independent deployments at the same live keyspace or writable checkpoint prefix.

Authentication and access changes#

Storage access decisions are cached: allowed decisions default to 600 seconds and denied decisions to 30 seconds. These are authorization.cache_ttl_secs and authorization.cache_negative_ttl_secs. Long-lived streams check access when they open; do not assume that revoking a permission immediately disconnects existing sessions or subscriptions. Plan token expiry, access changes and client reconnection accordingly.

The current read/stat-based authorization limitation is described in Deployment. Keep the backing Storage Service’s token validation enabled.

Configuration changes#

Maintain the deployment’s complete values in its normal release-management workflow. Changing ConfigMap-backed chart settings triggers a rolling restart; existing clients must reconnect. Rotating a Secret used as an environment variable does not automatically restart the service, so arrange the restart through that same workflow. Renewable token files are reread without requiring a restart.

Validate the updated values before applying them:

helm lint ./live-edit-service -f ./live-edit-service/live-edit-values.yaml
helm template live-edit-service ./live-edit-service \
  --namespace storage-apis -f ./live-edit-service/live-edit-values.yaml

These commands run from the parent of the extracted chart, using the values file created during installation. Retain all additional values files and their order when validating or applying changes. Repeat the authenticated smoke tests after deployment and check stream reconnection and backup health. Restore configuration through the chart/release workflow when rolling back; reverting only a pod template does not restore its ConfigMap contents.

Troubleshooting#

Symptom	Check
Startup rejects checkpoint configuration	Select a real adapter or explicitly disable checkpoints after establishing backing-store durability.
Pod stays unready	Inspect service and token-provider logs, Valkey connectivity, startup recovery progress and token validity. Preserve startup-probe allowance while diagnosing.
Authenticated operations fail	Verify the existing base asset, Storage access and token audience/expiry; verify User Info when enabled.
Layer subscriptions wait or time out	Check subscriber capacity, load distribution, Valkey latency and ingress timeouts.
Writes fail as the live store fills	Check memory against its configured limit; increase capacity or reduce retained workload. Keep `noeviction`.
Backups fail or remain pending	Verify service-token renewal, permissions, backup prefix and network access to Storage and transfer destinations.

Inspect logs without changing the running deployment:

kubectl logs deployment/live-edit-service -n storage-apis -c live-edit-service --tail=100
kubectl describe pods -n storage-apis -l app.kubernetes.io/instance=live-edit-service