Is this page helpful?

Observability#

Permission Service is instrumented end-to-end with OpenTelemetry for metrics and traces, and uses the Rust tracing framework for structured logs. This page describes what the service emits, how distributed tracing is wired up, which log levels are used, and how to configure all three through the Helm chart.

OpenTelemetry Overview#

Metrics and traces are exported using the OTLP protocol over gRPC to a user-supplied collector (for example, the OpenTelemetry Collector). Each exporter is configured independently:

Metrics are exported periodically from an SDK-managed PeriodicReader to the endpoint in observability.metrics.endpoint.
Traces are exported in batches from an SDK-managed batch exporter to the endpoint in observability.tracing.endpoint.

If either endpoint is left empty, the corresponding exporter is disabled and the data is dropped locally (no collector required). Logs are always written to stdout regardless of OpenTelemetry configuration.

All exported telemetry carries a common OpenTelemetry Resource populated from observability.serviceName (as service.name), the pod’s hostname (as service.instance.id), and the compiled-in crate version (as service.version), so data from individual pods can be distinguished in a multi-replica deployment and correlated with a specific build during rollouts or incident response.

Metrics#

The service exposes two kinds of metrics: service-specific metrics that describe authorization behavior and internal caches, and generic transport metrics that describe the REST and gRPC surfaces.

All metrics are OpenTelemetry instruments emitted through the global MeterProvider. Their names follow the OpenTelemetry semantic conventions where applicable.

Service-specific metrics#

These metrics capture the permission-service’s authorization decisions and the state of its in-memory caches. They are emitted from the service’s own instrumentation, not from any transport layer.

Name	Type	Unit	Attributes	Description
`authorization.request.allowed`	Counter	`requests`	—	Total number of authorization requests that resulted in an `allow` decision.
`authorization.request.denied`	Counter	`requests`	`deny_reason` = `explicit_forbid` \| `implicit` \| `cross_principal_denied` \| `evaluation_error`	Total number of authorization requests that resulted in a `deny` decision, categorized by why.
`evaluation.cache.size`	Gauge	—	—	Current number of entries held in the authorization-result LRU cache.
`evaluation.cache.hit`	Counter	`hits`	—	Evaluation-cache lookups that found a cached decision.
`evaluation.cache.miss`	Counter	`misses`	—	Evaluation-cache lookups that required a fresh policy evaluation.
`metadata.cache.size`	Gauge	—	—	Current number of entries held in the service-metadata LRU cache.
`metadata.cache.hit`	Counter	`hits`	—	Metadata-cache lookups that found cached metadata.
`metadata.cache.miss`	Counter	`misses`	—	Metadata-cache lookups that required a storage fetch.
`policy_set.cache.size`	Gauge	—	—	Current number of parsed Cedar `PolicySet` ASTs held in the policy-set LRU cache.
`policy_set.cache.hit`	Counter	`hits`	—	PolicySet-cache lookups that found a parsed Cedar AST.
`policy_set.cache.miss`	Counter	`misses`	—	PolicySet-cache lookups that required parsing policy text.
`token.cache.size`	Gauge	—	—	Current number of validated access tokens and principals held in the token cache.
`service.build_info`	Gauge	—	`version`	Always `1`. Presence plus the `version` attribute lets dashboards join other metrics by deployed version.

The cache gauges are refreshed on every scrape, so their sampled values reflect live cache utilization. The evaluation-cache, metadata-cache, and policy-set-cache hit/miss counters are incremented on every lookup and can be combined into a hit ratio (hit / (hit + miss)) to gauge cache effectiveness.

deny_reason distinguishes four structurally different deny paths: explicit_forbid (a policy explicitly denied the request), implicit (no policy matched at all), cross_principal_denied (a cross-principal permissions:view check failed), and evaluation_error (an internal evaluation error was converted into a fail-closed deny on the batch path). It is always one of these four fixed values, never free-text from a policy body or error message.

Policy evaluation metrics#

These metrics capture how much work a single uncached authorization decision required — see How authorization requests are evaluated for the order/priority evaluation model that order_groups_scanned refers to. Because an evaluation-cache hit skips policy retrieval and Cedar evaluation entirely, these histograms only reflect the cost of cache misses, which is the case operators most need to diagnose.

Name	Type	Unit	Attributes	Description
`authorization.evaluation.policies_evaluated`	Histogram	`policies`	`service`	Number of policies passed into the evaluator for a single action evaluation.
`authorization.evaluation.order_groups_scanned`	Histogram	`groups`	`service`	Number of distinct `order` groups iterated before a decision was reached (short-circuit depth).
`authorization.policy_fetch.size`	Histogram	`policies`	`service`, `source` = `single` \| `batch`	Number of policies returned by a single storage fetch.

service is the target service name from the evaluated action (action.service), a small, bounded-cardinality value. source distinguishes the single-action fetch path (authorize, diagnostics_authorize, meta-permission checks) from the batched per-(service, principal) fetch path used by authorize_batch.

Policy management metrics#

These metrics give a business-level view of policy create/update/delete activity — separate from the generic database metrics, which only describe storage-call health, not whether administrators are submitting policies that fail validation or lack permissions:edit. See Managing policies through the REST API for the endpoints these cover.

Name	Type	Unit	Attributes	Description
`policy.mutation.count`	Counter	`policies`	`operation` = `create` \| `update` \| `delete` \| `batch`, `outcome` = `success` \| `invalid` \| `forbidden` \| `error`	Every policy create/update/delete attempt, by outcome.
`policy.mutation.batch_size`	Histogram	`policies`	—	Number of policy items submitted in a single `PUT /v1beta/policies/batch/` call.

operation="batch" is only used for a whole-batch failure that occurs before per-item results exist (a request-level validation failure, or a permission check that fails for one item and aborts the whole batch, per the all-or-nothing semantics of PUT /v1beta/policies/batch/). Successful batch items are recorded individually as operation="create" / operation="update", so batch and single-item activity roll up into the same counts. outcome="forbidden" also covers policy mutations rejected for lacking permissions:edit, which otherwise produce no metric signal on the CRUD surface.

Notification metrics#

These metrics cover the outcome of publishing policy-change and metadata-change events to the Notification Service (Event Aggregation Service), and consuming them back for cross-replica cache invalidation — see Service notifications for when events are emitted and what they contain. The publish metrics are only recorded when an event publisher is configured (notifications.publisher.enabled: true); the consume metrics only when the consumer is configured (notifications.consumer.enabled: true).

Name	Type	Unit	Attributes	Description
`notification.publish.count`	Counter	`events`	`outcome` = `success` \| `retried_success` \| `failure` \| `retried_failure`; `event_type` = `omni.permissions.changed` \| `omni.permissions.meta.changed`	Outcome of each publish attempt, including the retry-after-token-refresh case. `event_type` distinguishes policy-change from metadata-change events.
`notification.publish.duration`	Histogram	`ms`	—	Wall-clock time from the first publish attempt to the final outcome (including any retry).

retried_success and retried_failure cover the case where the first attempt received Unauthenticated, the client-credentials token was refreshed, and the publish was retried once — see When events are published. A rising rate of retried_success is a useful early-warning signal even though it isn’t user-visible as an error: it usually means the client-credentials token is expiring earlier than expected, or being invalidated server-side.

The following metrics cover the built-in cross-replica cache-invalidation consumer — see Consuming change events for cross-replica cache invalidation. They are only recorded when the consumer is configured (notifications.consumer.enabled: true).

Name	Type	Unit	Attributes	Description
`notification.consume.count`	Counter	`events`	`event_type` = `omni.permissions.changed` \| `omni.permissions.meta.changed`; `outcome` = `invalidated_targeted` \| `invalidated_service` \| `invalidated_all` \| `invalidated_all_parse_error`	One per consumed event that triggered a local invalidation. `outcome` records whether a targeted key, a service, or the whole cache was invalidated (the `parse_error` variant is a full-flush fallback).
`notification.consume.errors`	Counter	`errors`	`reason` = `unauthenticated` \| `transport` \| `stream_closed`	Recorded whenever the subscription stream ends or errors.
`notification.consume.reconnect.count`	Counter	`reconnects`	`reason` = `initial` \| `token_expiry` \| `unauthenticated` \| `error` \| `stream_end`	Recorded immediately before each (re)open of the subscription stream.
`notification.consume.connected`	Gauge	—	—	`1` while this replica has an active subscription, `0` otherwise. A replica stuck at `0` silently goes stale until cache TTL — a good alerting signal.
`notification.consume.delivery_lag`	Histogram	`ms`	—	Delay between an event being published and this replica consuming it (`now - published_at`), i.e. end-to-end cross-replica propagation delay.

Generic transport metrics#

These metrics are emitted by the REST and gRPC middleware layers for every request the service handles. They are useful for general request-rate, latency, and error-rate dashboards regardless of which authorization endpoint is called.

REST metrics#

The following REST metrics are emitted by an internal middleware and are attributed with the matched route path, the HTTP method, and the resolved principal (empty when authentication is disabled or the caller is unauthenticated). The error counter additionally carries the status_code attribute:

Name	Type	Unit	Description
`http.server.request.count`	Counter	`requests`	Total number of HTTP requests received.
`http.server.request.duration`	Histogram	`ms`	End-to-end latency of HTTP request handling.
`http.server.request.errors`	Counter	`errors`	Total number of HTTP requests that returned a 4xx or 5xx status code.

http.server.request.active (UpDownCounter, unit requests, no attributes) tracks the number of HTTP requests currently being handled, incremented when a request enters the middleware and decremented when it finishes or its future is dropped early (for example on client disconnect). Unlike the metrics above, it reflects concurrency and saturation rather than throughput — a rising, sustained value indicates the service is handling more requests at once than it can drain, which is a useful signal for sizing SERVICE_WORKER_THREADS or the database connection pool, or for alerting before request queuing turns into timeouts.

In addition, the service mounts the axum-observability server metric layer, which emits metrics that follow the OpenTelemetry HTTP semantic conventions. These are attributed with http.request.method, http.route (when a matched path is available), http.response.status_code (for the duration metric), and network.protocol.version:

Name	Type	Unit	Description
`http.server.request.duration`	Histogram	`s`	Duration of inbound HTTP server requests, measured from call entry to the start of the response.
`http.server.request.body.size`	Histogram	`By`	Size of the request body consumed by the handler (headers and trailers excluded).
`http.server.response.body.size`	Histogram	`By`	Size of the response body observed by the client (headers and trailers excluded).

gRPC metrics#

The following metrics are emitted by an internal gRPC middleware and are attributed with rpc.service, rpc.method, and the final rpc.grpc.status_code. The naming follows the OpenTelemetry RPC semantic conventions:

Name	Type	Unit	Description
`rpc.server.request.count`	Counter	`requests`	Total number of gRPC server requests received.
`rpc.server.duration`	Histogram	`ms`	Duration of gRPC request handling.
`rpc.server.request.error.count`	Counter	`requests`	Total number of gRPC requests that completed with a non-OK status (non-zero `grpc-status`, or a transport failure reported as `UNKNOWN`).

rpc.server.request.active (UpDownCounter, unit requests, no attributes) tracks the number of gRPC requests currently being handled, following the same increment-on-start / decrement-on-drop pattern as http.server.request.active, and is likewise a concurrency/saturation signal rather than a throughput counter.

Tracing#

Tracing is built on the Rust tracing ecosystem, bridged to OpenTelemetry via tracing-opentelemetry. Spans are created either implicitly by #[tracing::instrument] annotations on handler functions or explicitly by the transport layer middlewares.

For each inbound REST request the TraceLayer middleware creates an http_request span with the HTTP method, the matched route, and OpenTelemetry attributes (otel.name = "<METHOD> <ROUTE>", otel.kind = "server"). For each inbound gRPC request a grpc_request span is created with otel.name = "<path>", otel.kind = "server", and a trace_id field populated from the current OpenTelemetry context.

Child spans are emitted for individual handler functions (policy and metadata CRUD, authorization middleware, JWT validation, userinfo lookups, token caching, event publishing, and so on) so that each stage of request processing is visible in the trace tree.

Distributed tracing#

Distributed tracing is supported. The service configures the W3C Trace Context propagator as the global OpenTelemetry text-map propagator, which means:

When an incoming request arrives (either REST or gRPC), the service extracts the parent trace context from the standard traceparent / tracestate HTTP headers and makes it the parent of the server-side span. If no valid context is present, a new root trace is started.
Trace IDs flow unchanged across the REST and gRPC surfaces, so a single client request that spans multiple services shows up as one end-to-end trace in the collector.

Callers that initiate traces upstream (for example, an API gateway, a client SDK, or another Omniverse service) only need to propagate the traceparent header for their traces to stitch together with Permission Service spans.

Sampling is driven by a TraceIdRatioBased sampler. The sampling ratio is taken from observability.tracing.samplingRatio (a value between 0.0 and 1.0); 1.0 samples every trace, 0.0 samples none.

Logging#

Logs are emitted through tracing and written to stdout. Log filtering uses the standard tracing_subscriber::EnvFilter directives syntax (the same syntax used by RUST_LOG), so the log level can be set globally or per target — for example:

info — enable info and above for every target.
permission_service=debug,hyper=warn — debug logs for the service itself, warnings only for hyper.

Two output formats are supported and are chosen at startup:

Pretty (default) — human-readable, colorized, multi-line output for interactive and development use.
Structured — single-line JSON records produced by tracing-subscriber’s built-in JSON formatter, suitable for ingestion by log aggregators. Enabled by setting observability.logging.structured.enabled to true.

Standard log-crate records from dependencies are forwarded into the same pipeline, so third-party libraries honor the configured level and format.

Log levels#

The service uses the five standard tracing levels with the following conventions:

error — Unrecoverable or user-visible failures. Examples: failure to initialize authentication on startup, failure to verify a user JWT, failure to fetch or parse JWKS or userinfo responses, failure to refresh the service-identity access token, invalid access-token files, and malformed JWT headers or claims. These messages indicate a real problem to investigate.
warn — Degraded but non-fatal conditions. Examples: authentication disabled via DISABLE_AUTH, missing or non-Bearer Authorization headers, non-2xx responses from the identity provider’s JWKS or userinfo endpoints, missing jwks_uri on the discovery document, failure to connect to the Event Aggregation Service, and failures to publish policy-change events.
info — Normal lifecycle events and operator-visible state changes. Examples: the effective service configuration at startup, “Listening HTTP on …” and “Listening gRPC on …” bind confirmations, successful JWKS pulls, assignment of system policies and system metadata, successful connection to the Event Aggregation Service, access-token file reads, and scheduled token refreshes. Each instrumented handler also emits info-level spans for incoming requests.
debug — Verbose per-request diagnostics useful during troubleshooting. Examples: absence of an Authorization header, number of policies matched for an action, request payloads for outgoing notification events, successful publication of policy-change events, and successful refreshes of client-credentials tokens.
trace — Reserved for very fine-grained diagnostics. The service itself does not currently emit trace-level records, but the level is wired up and can be enabled to pull trace-level output from dependencies through RUST_LOG-style directives.

Unless a specific subsystem needs a deeper view, info is the recommended level for steady-state operation and permission_service=debug is the recommended level for focused troubleshooting of the service itself.

Configuring Observability with the Helm Chart#

All observability settings live under the observability key in values.yaml. The chart translates these values into environment variables on the deployment (RUST_LOG, STRUCTURED_LOGGING, TRACES_ENDPOINT, TRACE_SAMPLING_RATIO, METRICS_ENDPOINT, SERVICE_NAME, SERVICE_SCOPE).

Helm values reference#

Value	Default	Description
`observability.serviceName`	`"permission-service"`	`service.name` resource attribute attached to all exported metrics and traces, and used as the tracer/meter name.
`observability.serviceScope`	`"permission_service"`	Instrumentation scope name used on exported metrics.
`observability.logging.level`	`"info"`	Log level filter (maps to `RUST_LOG`). Supports the full tracing directives syntax, for example `info` or `permission_service=debug,hyper=warn`.
`observability.logging.structured.enabled`	`false`	Enables JSON-structured log output produced by `tracing-subscriber`’s JSON formatter. Leave disabled for human-readable pretty logs.
`observability.tracing.endpoint`	`""`	OTLP/gRPC endpoint that receives trace spans, for example `http://otel-collector:4317`. Leaving this empty disables trace export.
`observability.tracing.samplingRatio`	`"1.0"`	Head-based sampling ratio between `0.0` and `1.0`. `1.0` samples every trace; lower values proportionally reduce the exported volume.
`observability.metrics.endpoint`	`""`	OTLP/gRPC endpoint that receives metrics, for example `http://otel-collector:4317`. Leaving this empty disables metrics export.

Example: export metrics, traces, and structured logs#

The following values block points both exporters at an OpenTelemetry Collector reachable inside the cluster, raises the service log level to debug, and switches log output to JSON so that a log pipeline can parse it:

observability:
  serviceName: "permission-service"
  serviceScope: "permission_service"

  logging:
    level: "permission_service=debug,info"
    structured:
      enabled: true

  tracing:
    endpoint: "http://otel-collector.observability.svc.cluster.local:4317"
    samplingRatio: "0.1"

  metrics:
    endpoint: "http://otel-collector.observability.svc.cluster.local:4317"

Additional environment variables#

The OpenTelemetry Rust SDK also honors its standard environment variables (for example OTEL_EXPORTER_OTLP_HEADERS, OTEL_RESOURCE_ATTRIBUTES, or OTEL_SDK_DISABLED). These can be injected through the chart’s extraVars map when a deployment needs to authenticate to a managed collector or add extra resource attributes:

extraVars:
  OTEL_EXPORTER_OTLP_HEADERS: "authorization=Bearer <token>"
  OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=production"