Observability (Metrics, Traces, Logs)#

This document describes Helm configuration and metric definitions for each service. Storage Service includes the Storage Core component; Event Aggregation Service and Event Consumer Service use the shared Notification Common library for telemetry.

Part I – Helm configuration by service#

Storage Service#

Helm / behavior	Details
Metrics	`OTEL_METRICS_EXPORTER` is set to `prometheus`. `OTEL_EXPORTER_PROMETHEUS_PORT` is set from `service.metrics.port`.
Ports	gRPC: `service.grpc.port` (default 8011). REST: `service.rest.port` (default 8012). Metrics: `service.metrics.port` (default 8013); Prometheus scrapes `GET /metrics` on this port.
Logging	`RUST_LOG` is built from `config.logging.level` and optional `config.logging.extra_targets`. `RUST_BACKTRACE` comes from `config.logging.backtrace`.
Traces	Not configured by default in Helm. OTLP tracing can be enabled by setting the appropriate OTEL environment variables (e.g. traces exporter and endpoint).
Logs	Logs go to stdout only.

Event Aggregation Service#

Enable metrics and tracing independently in event-aggregation-values.yaml:

observability:
  metrics:
    enabled: true
    endpoint:
      grpc: "http://otel-collector.observability.svc.cluster.local:4317"
  tracing:
    enabled: true
    endpoint:
      grpc: "http://otel-collector.observability.svc.cluster.local:4317"

Each enabled signal requires its endpoint.grpc value; a missing endpoint fails chart rendering. The endpoint.http fields are ignored. Log export is on when either signal is enabled, using the tracing endpoint or falling back to the metrics endpoint. logging.level sets the service log level.

When either signal is enabled, observability takes precedence over the legacy telemetry block. When both signals are disabled, telemetry.enabled and telemetry.otlp_*_endpoint remain in effect. Disable telemetry.enabled too if turning off export from a deployment that used the legacy settings.

gRPC authentication refusals are recorded as UNAUTHENTICATED; signing-key retrieval failures in jwt mode are INTERNAL. REST authentication refusals and oversized-body refusals return before request tracing starts, so they have no request span. Both are logged at WARNING. Check response results as well as HTTP status when monitoring publication: HTTP 200 can contain a failed publishing result. See Event Aggregation Operations for request limits, capacity and deployment checks.

Event Consumer Service#

Enable metrics and tracing independently in event-consumer-values.yaml:

observability:
  metrics:
    enabled: true
    endpoint:
      grpc: "http://otel-collector.observability.svc.cluster.local:4317"
  tracing:
    enabled: true
    endpoint:
      grpc: "http://otel-collector.observability.svc.cluster.local:4317"

Supply an OTLP/gRPC endpoint for each enabled signal. The endpoint.http fields are not used by this service. Log export is enabled whenever either signal is on; logs use the tracing endpoint, or the metrics endpoint when only metrics are enabled. logging.level controls the service log level.

If either observability signal is enabled, this block takes precedence over telemetry. If both are disabled, the legacy telemetry.enabled and telemetry.otlp_*_endpoint settings remain in effect. To disable export on a deployment that uses legacy settings, disable telemetry.enabled too.

For a per-pod capacity check, use GET /api/v1beta/metrics/channel-pool on the REST port. This JSON endpoint is independent of OTLP export and reports non-durable stream slots, despite its channels_in_use and channels_total field names. See Event Consumer Operations for scaling and health-probe guidance.

Non-durable logs, stream traces and cache_writer.* metrics identify a queue by queue_ref (sha256- followed by 12 hexadecimal characters), not the reconnect token. Update dashboards that used queue_id for those streams; durable queues still use queue_id. gRPC token refusals are recorded as UNAUTHENTICATED; JWKS-fetch failures in jwt mode are INTERNAL. REST authentication failures return 401 before request tracing starts, so those refusals have no request span.

Part II – Metric explanations by service#

Storage Service#

Common attributes include method, API version, storage backend, pod name, package version, and result.

Metric name	Type	Description
`storage.requests`	Counter	Total storage API requests
`storage.request.duration`	Histogram	Request duration (seconds)
`storage.sdk.requests`	Counter	Requests to the storage backend SDK
`storage.sdk.request.duration`	Histogram	Backend SDK call duration
`storage.enumeration.items`	Histogram (U64)	Items returned per enumeration (list, list_stat, enumerate, enumerate_versions)
`storage.read.redirects`	Counter	Redirect URLs returned for read operations
`storage.read.chunk.size`	Histogram (U64)	Chunk size from backend (bytes)
`storage.read.object.size`	Histogram (U64)	Total size of read objects (bytes)
`storage.write.operations`	Counter	Write operations by upload method (body, redirect, multipart)

gRPC server metrics#

Metric name	Type	Description
`grpc.server.call.started`	Counter	Server calls started
`grpc.server.call.rcvd_total_compressed_message_size`	Histogram (U64)	Compressed bytes received per RPC
`grpc.server.call.sent_total_compressed_message_size`	Histogram (U64)	Compressed bytes sent per call
`grpc.server.call.duration`	Histogram (f64)	Call duration (seconds). Attributes: grpc.method, grpc.service, grpc.name, grpc.status

Metadata cache#

Metric name	Type	Description
`storage_metadata_cache_cache_access`	Counter	Cache access count (attributes: pod, name)
`storage_metadata_cache_cache_miss`	Counter	Cache miss count
`storage_metadata_cache_entries`	Gauge	Current cache entry count
`storage_metadata_cache_size`	Gauge	Cache memory footprint (bytes)

Metadata backends#

S3: storage_s3_metadata_list_objects_latency, storage_s3_metadata_get_object_latency, storage_s3_metadata_put_object_latency, storage_s3_metadata_delete_object_latency (histograms).

DynamoDB: storage_aws_dynamodb_metadata_put_item_latency, storage_aws_dynamodb_metadata_delete_item_latency, storage_aws_dynamodb_metadata_batch_get_item_latency, storage_aws_dynamodb_metadata_batch_get_item_request_items, storage_aws_dynamodb_metadata_query_latency.

Azure Blob: storage_azure_blob_metadata_list_blobs_latency, storage_azure_blob_metadata_get_blob_latency, storage_azure_blob_metadata_put_blob_latency, storage_azure_blob_metadata_delete_blob_latency.

Azure Table: storage_azure_table_metadata_query_latency, storage_azure_table_metadata_query_response_size, storage_azure_table_metadata_delete_latency, storage_azure_table_metadata_update_latency, storage_azure_table_metadata_insert_or_update_latency.

HTTP status is also recorded per backend call (per status code).

Pub/sub#

SQS: storage_sqs_delete_message_batch_latency, storage_sqs_delete_message_batch_batch_size, storage_sqs_delete_message_batch_active_requests, storage_sqs_receive_message_latency, storage_sqs_receive_message_batch_size, storage_sqs_receive_message_active_requests, storage_sqs_events_per_message.

Azure Service Bus: storage_azure_service_bus_receive_messages_latency, storage_azure_service_bus_receive_messages_batch_size, storage_azure_service_bus_receive_messages_active_requests, storage_azure_service_bus_complete_message_latency, storage_azure_service_bus_complete_message_active_requests.

Notification client: storage_notification_service_publish_latency, storage_notification_service_publish_active_requests. OAuth2: storage_oauth2_client_credentials_provider_get_token_latency, storage_oauth2_client_credentials_provider_get_token_results.

Event Aggregation Service#

Uses the same gRPC server metrics as Event Consumer Service (Notification Common), listed in the Event Consumer Service section. It does not define additional service-specific metrics.

Event Consumer Service#

Notification Common gRPC metrics (shared with Event Aggregation Service)#

Metric name	Type	Description
`rpc.server.active_requests`	UpDownCounter	In-flight requests
`rpc.server.requests_per_rpc`	Histogram	Requests per RPC
`rpc.server.request_size`	Histogram	Request size
`rpc.server.response_size`	Histogram	Response size
`rpc.server.responses_per_rpc`	Histogram	Responses per RPC
`rpc.server.calls`	Counter	Total gRPC calls
`rpc.server.active_methods`	UpDownCounter	In-flight methods
`rpc.server.active_responses`	UpDownCounter	In-flight responses
`rpc.server.duration`	Histogram	RPC duration

Consumer metrics (Event Consumer Service only)#

Metric name	Type	Description
`events_by_type_total`	Counter	Events processed by event type (label: event_type)
`events_processed_successfully_total`	Counter	Successfully processed events
`events_processing_failed_total`	Counter	Events that failed during processing
`event_processing_duration_ms`	Histogram	Time spent processing each event (ms)

SSE metrics (Event Consumer Service only)#

Metric name	Type	Description
`sse.server.active_connections`	UpDownCounter	Active SSE connections
`sse.server.connections_total`	Counter	Total SSE connections established
`sse.server.connection_duration`	Histogram	Connection duration (ms)
`sse.server.events_sent_total`	Counter	Events sent via SSE
`sse.server.events_per_connection`	Histogram	Events sent per SSE connection
`sse.server.connection_errors_total`	Counter	SSE connection errors

Labels include endpoint and, where applicable, event_type.