Procedure

Create the NVCF Telemetry Endpoint

NVCF Configuration

Add telemetry endpoint (Azure Monitor example).

1. Create a new Application Insights instance in the Azure Portal (Monitor -> Application Insights -> + Create). Choose Subscription, Resource Group and a Log Analytics workspace. Select + create**.

2. Once created, open Overview -> JSON View and copy the ConnectionString value.

3. Create the telemetry endpoint in NVCF (UI):

   • Navigate to https://nvcf.ngc.nvidia.com/ -> Settings.

   • Under Telemetry Endpoints click + Add Endpoint.

   • Name the endpoint (for example, azure-monitor-endpoint).

   • Select Azure Monitor.

   • Paste values extracted from the Azure ConnectionString into the Telemetry endpoint fields:

     • Endpoint (IngestionEndpoint): https://xxxx-x.in.applicationinsights.azure.com/

     • Instrumentation Key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

     • LiveEndpoint: https://xxxx.livediagnostics.monitor.azure.com/

     • ApplicationId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

   • Select Logs and Metrics under Telemetry Type.

   • Select HTTP for the communication protocol.

   • Click Save Configuration.

4. Optionally create via the CLI (example):

curl -s --location --request POST 'https://api.ngc.nvidia.com/v2/nvcf/telemetries' \
 --header 'Content-Type: application/json' \
 --header 'Authorization: Bearer '$NVCF_TOKEN \
 --data '{
        "endpoint": "YOUR_AZURE_MONITOR_ENDPOINT",
        "protocol": "HTTP",
        "provider": "AZURE_MONITOR",
        "types": [
                "LOGS",
                "METRICS"
        ],
        "secret": {
                "name": "YOUR_NVCF_TELEMETRY_NAME",
                "value": {
                        "instrumentationKey": "YOUR_INSTRUMENTATION_KEY",
                        "liveEndpoint": "YOUR_LIVE_ENDPOINT",
                        "applicationId": "YOUR_APPLICATION_ID"
                }
        }
}'

When you attach a telemetry endpoint to your NVCF function, the platform automatically injects the following environment variables into your container:

• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT

• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT

For additional information, refer to the Observability Guide.

Create a telemetry ID

Before creating the function, ensure you have created a telemetry endpoint on NVCF. See the Platform Configuration for External Observability Setup Guide for the necessary configuration documentation.

Retrieve the Telemetry ID

Capture the telemetryId for use in the function creation using this command:

curl -s --location --request GET 'https://api.ngc.nvidia.com/v2/nvcf/telemetries' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer '$NVCF_TOKEN'' | jq

Store the Telemetry ID

Export the telemetry ID as an environment variable on your workstation for use in the function creation command:

export TELEMETRY_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

This value is passed to the NGC API when creating the function (that is, Deploying Kit Apps). NVCF uses this ID to configure the telemetry routing at the platform level for your function.

Kit App Configuration Reference

The settings in this table are some of the more commonly used configuration options for OpenTelemetry use in Kit Apps:

Setting	Description	Default
–/observability/logs/emitLevel=<level>	Minimum log level to export via OTLP (verbose, info, warn, error, fatal)	warn
–/observability/logs/channels/<name>=<level>	Settings branch to provide channel-specific log-level filtering for OTLP (each entry’s name should be the name or wildcard pattern of a log channel, and its value should be the desired log level).	none
–/observability/logs/adoptLoggingChannelSettings=<bool>	Match the OTLP log filtering channel settings to the channel settings of the main Carbonite logging system if set to ‘true’.	false
–/exts/omni.kit.livestream.app/primaryStream/enableOpenTelemetry=true	Enable streaming metrics export (must be enabled to receive streaming metrics).	false

OTel Environment Variables

These are some of the more commonly used OTel environment variables that can help customize the behaviour of your Kit telemetry on your observability backend. These settings can help organize the telemetry in your observability backend and promote ease in filtering.

Variable	Description	Example
OTEL_SERVICE_NAME	Service name in telemetry signals	my-kit-app
OTEL_RESOURCE_ATTRIBUTES	Additional resource metadata comma-separated	environment=prod,version=1.0

Carbonite OTel Variables and Default Values

For a full listing of additional OTEL configuration variables and settings, refer to the Configuring Opentelemetry documentation.

Channel-Specific Filtering (Advanced)

To filter specific log channels, add to NVDA_KIT_ARGS:

--/observability/logs/emitLevel=warn
--/observability/logs/channels/omni.kit.*=error
--/observability/logs/channels/my.app.*=info

Create the function via CLI

When creating or updating your NVCF function, add the OTel configuration to containerEnvironment. The key settings are passed via NVDA_KIT_ARGS and NVCF Function Environment variables:

curl -s -v --location --request POST 'https://api.ngc.nvidia.com/v2/nvcf/functions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer '$NVCF_TOKEN'' \
--data '{
"name": "'${STREAMING_FUNCTION_NAME:-usd-composer}'",
"inferenceUrl": "'${STREAMING_START_ENDPOINT:-/sign_in}'",
"inferencePort": '${STREAMING_SERVER_PORT:-49100}',
"health": {
        "protocol": "HTTP",
        "uri": "/v1/streaming/ready",
        "port": '${CONTROL_SERVER_PORT:-8011}',
        "timeout": "PT10S",
        "expectedStatusCode": 200
},
"containerImage": "'$STREAMING_CONTAINER_IMAGE'",
"apiBodyFormat": "CUSTOM",
"description": "'${STREAMING_FUNCTION_NAME:-usd-composer}'",
"functionType": "STREAMING",
"containerEnvironment": [
        {"key": "NVDA_KIT_NUCLEUS", "value": "'$NUCLEUS_SERVER'"},
        {"key": "OMNI_JWT_ENABLED", "value": "1"},
        {"key": "OTEL_SERVICE_NAME", "value": "'${STREAMING_FUNCTION_NAME:-usd-composer}'"},
        {"key": "NVDA_KIT_ARGS", "value": "--/observability/logs/emitLevel=info --/exts/omni.kit.livestream.app/primaryStream/enableOpenTelemetry=true --/app/livestream/nvcf/sessionResumeTimeoutSeconds=300"}
],
"telemetries": {
        "logsTelemetryId": "'$TELEMETRY_ID'",
        "metricsTelemetryId": "'$TELEMETRY_ID'"
}
}'

Confirm Telemetry within the Observability Platform

Run the observability-specific queries to confirm the telemetry. Sample references are available here:

• Azure Monitor

• Grafana Cloud

Sample Kit Dashboards for Azure Monitor and Grafana Cloud are provided for additional reference.