Is this page helpful?

Kit Artifacts Collection#

../../../_images/ov_cloud_banner.jpg

Overview#

Kit Artifacts Collection automatically uploads Kit logs, profiler traces, and metadata to cloud storage when a containerized Kit application exits. This optional feature helps with debugging, troubleshooting, and performance analysis by preserving artifacts from your streaming sessions.

The artifact collector supports these Cloud Service Providers (CSPs):

  • Microsoft Azure (Azure Blob Storage)

  • AWS (S3)

Artifacts are uploaded on container exit (normal or signal-terminated) and include:

  • The most recent Kit log from ~/.nvidia-omniverse/logs/Kit/

  • Profiler traces in chrome tracing format (if present)

  • A metadata.json file with NVCF context, exit code, and file information

Configuration Options#

The artifact collector is configured via NVCF secrets or environment variables. All configuration must be set when creating the NVCF function.

Required Configuration#

Variable

Description

Example

ARTIFACT_STORAGE_URI

Cloud storage destination URI

s3://my-bucket, az://my-container

Optional Configuration#

Variable

Description

Default

ARTIFACT_PATH_TEMPLATE

Path template with variable substitution

{NVCF_FUNCTION_NAME}/{session_id}

ARTIFACT_MAX_FILE_SIZE_MB

Maximum file size in MB for uploads

512

ARTIFACT_UPLOAD_ON_ERROR_ONLY

Upload artifacts only when exit code is non-zero

false

Cloud Provider Credentials#

Azure Blob Storage#

ARTIFACT_STORAGE_URI=az://my-container
AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...

AWS S3#

ARTIFACT_STORAGE_URI=s3://my-bucket
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=...

Path Templates#

The ARTIFACT_PATH_TEMPLATE uses Python format syntax to organize uploaded artifacts. You can use NVCF environment variables and helper variables to create structured paths.

Available Variables#

NVCF Environment Variables#

All NVCF variables are automatically available:

Variable

Description

{NVCF_FUNCTION_ID}

Unique function identifier

{NVCF_FUNCTION_NAME}

Function name

{NVCF_FUNCTION_VERSION_ID}

Function version identifier

{NVCF_NCA_ID}

Organization’s NCA ID

{NVCF_BACKEND}

Backend/cluster group

{NVCF_INSTANCETYPE}

Instance type

{NVCF_REGION}

Deployment region

{NVCF_ENV}

Spot environment (if applicable)

{NVCF_REQID}

Request ID (also used as session_id)

Helper Variables#

Variable

Description

Example

{session_id}

NVCF request ID or fallback identifier

abc-123-def-456

{date}

Current date

2025-12-09

{datetime}

Current date and time

2025-12-09_14-30-15

{hostname}

Container hostname

nvcf-pod-12345

Note

Understanding session_id vs. NVCF Request ID

The {session_id} variable used in artifact path templates corresponds to the NVCF request ID ({NVCF_REQID}), which is automatically available as an environment variable within the NVCF container. This is not the same as the streaming session ID returned by the POST /stream API endpoint.

For streaming functions:

  • The streaming session ID is what users receive from the POST /stream endpoint (e.g., "id": "82a67a28-66e2-4f01-bdf8-2a20971f2457")

  • The NVCF request ID is the underlying NVCF function invocation identifier, available as NVCF_REQID within the container

Retrieving NVCF Request ID from Streaming Session ID

If you’re using the Portal Sample backend, you can retrieve the NVCF request ID for a streaming session using the backend API:

curl -X GET "https://<portal_url>/api/sessions/{session_id}" \
  -H "Authorization: Bearer <token>"

The response includes NVCF metadata such as nvcf_reqid (the NVCF request ID), nvcf_function_id, and nvcf_function_version_id. Use the nvcf_reqid value to locate the corresponding artifacts in your cloud storage bucket.

Alternatively, consider logging both IDs in your application or including custom metadata in your Kit application logs for easier correlation.

Path Template Examples#

# Default (simple organization by function name)
ARTIFACT_PATH_TEMPLATE={NVCF_FUNCTION_NAME}/{session_id}

# Multi-level organization
ARTIFACT_PATH_TEMPLATE={NVCF_FUNCTION_ID}/{NVCF_FUNCTION_VERSION_ID}/{NVCF_REGION}/{session_id}

# Project-based organization
ARTIFACT_PATH_TEMPLATE=my-project/prod/{NVCF_REGION}/{datetime}

# Simple session ID
ARTIFACT_PATH_TEMPLATE={session_id}

Complete Example#

This example shows a complete configuration using AWS S3 with structured paths:

# Required
ARTIFACT_STORAGE_URI=s3://my-kit-artifacts-bucket

# Optional
ARTIFACT_PATH_TEMPLATE={NVCF_FUNCTION_NAME}/v{NVCF_FUNCTION_VERSION_ID}/{NVCF_REGION}/{date}/{session_id}
ARTIFACT_MAX_FILE_SIZE_MB=512  # Skip files larger than 512 MB
ARTIFACT_UPLOAD_ON_ERROR_ONLY=false  # Set to true to only upload on errors/failures

# AWS credentials
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=...

This configuration uploads Kit logs, profiler traces, and metadata to structured paths like:

s3://my-kit-artifacts-bucket/my-function/v123/my-default-region/2025-12-09/req-789xyz/
├── metadata.json
├── kit_20251209_143015.log
└── ct-profile_2025-12-09_14-30-45.json.gz

Adding Secrets to NVCF Functions#

To enable artifact collection, you must pass the required configuration as secrets when creating your NVCF function. The Portal Sample’s create_function.sh script does not currently support secrets, so you’ll need to modify it or create the function manually via the NVCF API.

Modifying create_function.sh#

The Portal Sample provides a create_function.sh script for creating NVCF functions. To add artifact collection support, modify the script to include a secrets section in the API request.

Here’s an example of the modified API call with artifact collection secrets (AWS S3 example):

nvcf_creation_response=$(curl -s -v --location --request POST 'https://api.ngc.nvidia.com/v2/nvcf/functions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer '$NVCF_TOKEN'' \
--data '{
  "name": "'${STREAMING_FUNCTION_NAME:-usd-composer}'",
  "inferenceUrl": "'${STREAMING_START_ENDPOINT:-/sign_in}'",
  "inferencePort": '${STREAMING_SERVER_PORT:-49100}',
  "health": {
    "protocol": "HTTP",
    "uri": "/v1/streaming/ready",
    "port": '${CONTROL_SERVER_PORT:-8111}',
    "timeout": "PT10S",
    "expectedStatusCode": 200
  },
  "containerImage": "'$STREAMING_CONTAINER_IMAGE'",
  "apiBodyFormat": "CUSTOM",
  "description": "'${STREAMING_FUNCTION_NAME:-usd-composer}'",
  "functionType": "STREAMING",
  "containerEnvironment": [
    {"key": "NVDA_KIT_NUCLEUS", "value": "'$NUCLEUS_SERVER'"},
    {"key": "OMNI_JWT_ENABLED", "value": "1"},
    {"key": "NVDA_KIT_ARGS", "value": "--/app/livestream/nvcf/sessionResumeTimeoutSeconds=300"}
  ],
  "secrets": [
    {"name": "ARTIFACT_STORAGE_URI", "value": "s3://my-kit-artifacts-bucket"},
    {"name": "ARTIFACT_PATH_TEMPLATE", "value": "{NVCF_FUNCTION_NAME}/{session_id}"},
    {"name": "ARTIFACT_MAX_FILE_SIZE_MB", "value": "512"},
    {"name": "AWS_ACCESS_KEY_ID", "value": "'$AWS_ACCESS_KEY_ID'"},
    {"name": "AWS_SECRET_ACCESS_KEY", "value": "'$AWS_SECRET_ACCESS_KEY'"},
    {"name": "AWS_DEFAULT_REGION", "value": "us-west-2"}
  ]
}
')

Azure Blob Storage Example#

For Azure Blob Storage, use these secrets:

"secrets": [
  {"name": "ARTIFACT_STORAGE_URI", "value": "az://my-container"},
  {"name": "ARTIFACT_PATH_TEMPLATE", "value": "{NVCF_FUNCTION_NAME}/{session_id}"},
  {"name": "AZURE_STORAGE_CONNECTION_STRING", "value": "DefaultEndpointsProtocol=https;AccountName=..."}
]

Creating Functions via NVCF UI#

Alternatively, you can create functions using the NVCF UI. When creating or updating a function:

  1. Navigate to the Secrets section

  2. Click + Add Secret for each required configuration variable

  3. Enter the secret name and value

  4. Save the function configuration

Important Notes and Limitations#

  • Artifacts are uploaded only on container exit (normal or signal-terminated)

  • Files exceeding the size limit (default 512 MB) are skipped and recorded in metadata.json

  • Upload failures are logged but don’t prevent container exit

  • Check container logs for [ARTIFACT_COLLECTOR] messages to verify operation

  • The artifact collector works independently of the Vector logging solution

  • Ensure your cloud storage credentials have write permissions to the specified bucket/container

  • When ARTIFACT_UPLOAD_ON_ERROR_ONLY is enabled:

    • Uploads occur only when exit code is non-zero (errors, crashes, or signal termination)

    • Exit code 0 (successful completion) skips artifact upload to save storage costs

    • Signal termination (SIGTERM, SIGINT) results in non-zero exit codes and triggers upload