Procedure

Validate Local Container

Verify the Kit App appears in the list of Docker images before proceeding:

docker image ls

Note that this page uses kit_app_template:latest as the base Kit container. If you use a different name for the Kit container, note it — the name is required when working with the Dockerfile.

Create Working Directory

In your Ubuntu terminal run:

mkdir kitvector
cd kitvector

Integrate Vector Log Processor with Kit Container

This section covers the integration of Vector.dev log processor into the existing Kit Application container. The integration involves creating a custom entrypoint script, defining Vector OTEL transforms, and building a new Docker image that includes both the Kit application and Vector.

Configure Vector with a Custom Entrypoint Script

Create entrypoint_vector_dev.sh in your working directory:

touch entrypoint_vector_dev.sh

Script Components Overview

The custom entrypoint script orchestrates Vector configuration and application startup. It handles:

• Environment setup (user environment, debug logging, envvar validation)

• Vector OTel processing control (checks VECTOR_OTEL_ACTIVE to enable/disable processing)

• Configuration management (support for VECTOR_CONF_B64 or static configuration; OTEL endpoint substitution)

• Validation & testing (Vector config validation and OTel endpoint connectivity checks)

• Process orchestration (start Vector, launch Kit, pipe Kit output to log files)

Add the Script Content

Edit the entrypoint script:

nano entrypoint_vector_dev.sh

Paste the following into entrypoint_vector_dev.sh:

#!/bin/bash

echo "[entrypoint_vector_dev.sh] Starting container..."

# Set required environment variables for Kit
export USER="ubuntu"
export LOGNAME="ubuntu"

# Check if Vector OTEL processing is enabled
if [ "$VECTOR_OTEL_ACTIVE" = "TRUE" ]; then
        echo "[Vector] Vector OTEL processing is ENABLED (VECTOR_OTEL_ACTIVE=TRUE)"

        # Create log files if they do not exist
        echo "[Vector] Setting up log files..."
        touch /tmp/kit_structured_logs.log
        chmod 666 /tmp/kit_structured_logs.log
 # Validate OTEL endpoint
        if [ -z "$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT" ]; then
                echo "[Vector] ERROR: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT is not set!"
                exit 1
        fi

        if [[ ! "$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT" =~ ^https?:// ]]; then
                echo "[Vector] ERROR: Invalid OTEL endpoint format. Must start with http:// or https://"
                exit 1
        fi

        echo "[Vector] Using OTEL endpoint: $OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"

        # Determine which Vector configuration to use
        if [ ! -z "$VECTOR_CONF_B64" ]; then
                echo "[Vector] Custom Vector configuration provided via VECTOR_CONF_B64"
                echo "[Vector] Decoding and using customer-provided configuration..."

                # Decode Vector config
                echo "$VECTOR_CONF_B64" | base64 -d > /tmp/vector_raw.toml

                # Replace OTEL endpoint
                sed "s|PLACEHOLDER_OTEL_ENDPOINT|$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT|g" /tmp/vector_raw.toml > /tmp/vector.toml

                echo "[Vector] Using CUSTOM Vector configuration (from VECTOR_CONF_B64)"
        else
                echo "[Vector] No custom configuration provided. Using static/default Vector configuration..."

                # Copy static configuration and replace OTEL endpoint
                cp /opt/vector/static_config.toml /tmp/vector_raw.toml
                sed "s|PLACEHOLDER_OTEL_ENDPOINT|$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT|g" /tmp/vector_raw.toml > /tmp/vector.toml
                echo "[Vector] Using STATIC Vector configuration (from /opt/vector/static_config.toml)"
        fi

        # Validate Vector config
        echo "[Vector] Verifying Vector configuration..."
        if [ -x "/opt/vector/bin/vector" ]; then
                /opt/vector/bin/vector validate /tmp/vector.toml
                if [ $? -ne 0 ]; then
                        echo "[Vector] ERROR: Vector configuration validation failed!"
                        exit 1
                fi
        fi

        echo "[Vector] Starting Kit with real-time log forwarding..."

        # Start Vector reading from the log file
        /opt/vector/bin/vector --config /tmp/vector.toml &

        # Start Kit and redirect its output to the log file (no stdout)
        stdbuf -oL /entrypoint.sh >> /tmp/kit_structured_logs.log 2>&1 &

        # Wait for the Kit process to appear
        while true; do
                KIT_PID=$(pgrep -n "kit")
                if [ -n "$KIT_PID" ]; then
                        echo "Process Kit has started with PID: $KIT_PID"
                        break
                fi
                sleep 1  # Check every second
        done

        # Monitor Kit process until it ends
        echo "[Vector] Monitoring Kit process..."
        while kill -0 $KIT_PID 2>/dev/null; do
                sleep 1
        done

        echo "[Vector] Kit pipeline has ended"
        echo "[Vector] Cleaning up processes..."

        # Kill all processes owned by current user to ensure clean shutdown
        echo "[Vector] Killing all user processes..."
        killall -u $USER 2>/dev/null || true

        echo "[Vector] Container exiting"
        exit 0
else
        echo "[Vector] Vector OTEL processing is DISABLED (VECTOR_OTEL_ACTIVE=FALSE or not set)"
        echo "[Vector] Running Kit without log processing."
        exec /entrypoint.sh
fi

Vector OTel Transform

The Vector OTel Transform converts Kit application plaintext logs into OpenTelemetry (OTel) format required by NVCF. Vector uses the Vector Remap Language (VRL) to perform transformations; extend the VRL as needed.

Transformation process:

• Input: Kit application plaintext logs (stdout/stderr)

• Processing: Vector Remap Language (VRL) transformation rules

• Output: OpenTelemetry-compliant JSON logs

For the purpose of this page, we are using verified VRL syntax to work appropriately with the log processing pipeline.

The vector.toml file being created as part of this step, and will serve as the static configuration file. This step is provided as a reference to how the VRL works. This is a required step as part of the configuration.

To pass a custom vector configuration, use the VECTOR_CONF_B64 environment variable.

Create the Vector Configuration File

Create and edit vector.toml:

touch vector.toml
nano vector.toml

Example vector.toml (copy into the file):

[sources.kit_logs]
type = "file"
include = ["/tmp/kit_structured_logs.log"]
read_from = "beginning"
max_line_bytes = 1024
ignore_older_secs = 86400  # Ignore logs older than 24 hours
remove_after_secs = 604800  # Remove processed files after 7 days

[transforms.otel_transforms]
type = "remap"
inputs = ["kit_logs"]
source = '''

# Extract log level from message
level = "INFO"
message = to_string(.message) ?? ""
if message != "" {
   if match(message, r'\[Error\]') {
      level = "ERROR"
   } else if match(message, r'\[Warning\]') {
      level = "WARN"
   } else if match(message, r'\[Debug\]') {
      level = "DEBUG"
   }
}

# Set severity number based on level
severity_number = 9
if level == "ERROR" {
   severity_number = 17
} else if level == "WARN" {
   severity_number = 13
} else if level == "DEBUG" {
   severity_number = 5
}
.resourceLogs = [{
   "resource": {"attributes": [{
      "key": "service.name",
      "value": {"stringValue": "kit-vector-app"}
   }]
   },
   "scopeLogs": [{
      "scope": {
         "name": "kitvector.log",
         "version": "1.0.0"
      },
      "logRecords": [{
         "timeUnixNano": to_string(to_unix_timestamp(now(), unit: "nanoseconds")),
         "body": {"stringValue": .message},
         "severityText": level,
         "severityNumber": severity_number
      }]
   }]
}]
'''
[sinks.otel_collector]
type = "http"
inputs = ["otel_transforms"]
uri = "${OTEL_EXPORTER_OTLP_LOGS_ENDPOINT}"
encoding.codec = "json"
method = "post"
request.headers.Content-Type = "application/json"
framing.method = "newline_delimited"
batch.max_events = 1
batch.timeout_secs = 1
request.retry_attempts = 3
request.retry_initial_backoff_secs = 1
request.retry_max_duration_secs = 10

This configuration allows kit logs to be read from the stdout/stderr redirected log file /tmp/kit_structured_logs.log.

The OTel transformed logs are directed to the NVCF’s OTEL_EXPORTER_OTLP_LOGS_ENDPOINT and to the console. As a result, you can observe the Kit application logs on the NVCF UI and your observability backend.

Create the Dockerfile

Create and edit Dockerfile:

touch Dockerfile
nano Dockerfile

Copy the following into the Dockerfile:

FROM kit_app_template:latest

USER root

RUN apt-get update && \
    apt-get install -y curl && \
    mkdir -p /opt/vector && \
    curl -L https://packages.timber.io/vector/0.46.1/vector-0.46.1-x86_64-unknown-linux-gnu.tar.gz -o /tmp/vector.tar.gz && \
    tar -xzf /tmp/vector.tar.gz -C /opt/vector --strip-components=2 && \
    rm -rf /tmp/vector*

RUN mkdir -p /logs

# Ensure ubuntu home directory exists for NVCF compatibility (user already exists in base image)
RUN mkdir -p /home/ubuntu && \
    chown -R ubuntu:ubuntu /home/ubuntu

# Create Vector data directory and give ubuntu user access
RUN mkdir -p /var/lib/vector && \
    chown -R ubuntu:ubuntu /var/lib/vector

COPY entrypoint_vector_dev.sh /entrypoint_vector_dev.sh
COPY vector.toml /opt/vector/static_config.toml
RUN chmod +x /entrypoint.sh /entrypoint_vector_dev.sh

# Switch back to ubuntu user for runtime
USER ubuntu

ENTRYPOINT ["/entrypoint_vector_dev.sh"]

Build the Kit Vector Container Image

Before building the container, verify that you have all the necessary files in the current directory:

ls -la

Build your enhanced Kit container with Vector integration:

docker build -t byoo_kit_vector:latest .

If you followed along the steps, the required files for the Dockerfile should have been created.

Verify the docker image:

docker image ls

The expected output should look like this:

REPOSITORY          TAG       IMAGE ID       CREATED        SIZE
byoo_kit_vector     latest    1234567890123   1 hour ago     7.67GB

To push your container to NGC for deployment, follow the steps here.

NVCF Configuration

Add telemetry endpoint (Azure Monitor example)

Create a new Application Insights instance in the Azure Portal (Monitor -> Application Insights -> + Create). Choose Subscription, Resource Group and a Log Analytics workspace, then Review + create. Once created, open Overview -> JSON View and copy the ConnectionString value.

Create the telemetry endpoint in NVCF (UI):

• Navigate to https://nvcf.ngc.nvidia.com/ -> Settings.

• Under Telemetry Endpoints click + Add Endpoint.

• Name the endpoint (e.g., azure-monitor-endpoint).

• Select Azure Monitor.

• Paste values extracted from the Azure ConnectionString into the Telemetry endpoint fields:

  • Endpoint (IngestionEndpoint): https://xxxx-x.in.applicationinsights.azure.com/

  • Instrumentation Key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

  • LiveEndpoint: https://xxxx.livediagnostics.monitor.azure.com/

  • ApplicationId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

• Select Logs and Metrics under Telemetry Type.

• Select HTTP for the communication protocol.

• Click Save Configuration.

Optionally create via the CLI (example):

curl -s --location --request POST 'https://api.ngc.nvidia.com/v2/nvcf/telemetries' \
 --header 'Content-Type: application/json' \
 --header 'Authorization: Bearer '$NVCF_TOKEN \
 --data '{
        "endpoint": "YOUR_AZURE_MONITOR_ENDPOINT",
        "protocol": "HTTP",
        "provider": "AZURE_MONITOR",
        "types": [
                "LOGS",
                "METRICS"
        ],
        "secret": {
                "name": "YOUR_NVCF_TELEMETRY_NAME",
                "value": {
                        "instrumentationKey": "YOUR_INSTRUMENTATION_KEY",
                        "liveEndpoint": "YOUR_LIVE_ENDPOINT",
                        "applicationId": "YOUR_APPLICATION_ID"
                }
        }
}'

Get Telemetry ID

Once the telemetry endpoint is created, capture its telemetryId (required for CLI-based function creation). This step is not required when using the NVCF UI:

echo NVCF_TOKEN="nvapi-xxxxxxxxxxxxxxxxxxxxxx"

curl -s --location --request GET 'https://api.ngc.nvidia.com/v2/nvcf/telemetries' \
 --header 'Content-Type: application/json' \
 --header 'Authorization: Bearer '$NVCF_TOKEN'' | jq

Copy the telemetryId for the created endpoint and store it:

export TELEMETRY_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Environment Variables

The implementation uses environment variables to control Vector behavior. Set these when deploying the NVCF function.

Key environment variables:

• VECTOR_OTEL_ACTIVE — TRUE | FALSE / unset - When TRUE: container uses Vector for log processing and forwarding to NVCF collector - When FALSE or unset: container bypasses Vector and runs Kit directly via /entrypoint.sh

• VECTOR_CONF_B64 — base64-encoded string - Provides a custom Vector configuration via base64-encoded string. If provided, the entrypoint decodes and uses it; otherwise the default static /opt/vector/static_config.toml is used.

To base64-encode a configuration file:

base64 -w 0 vector.toml

Container to Function Flow - Include Telemetry Endpoint

Create function via CLI:

curl -s -v --location --request POST 'https://api.ngc.nvidia.com/v2/nvcf/functions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer '$NVCF_TOKEN'' \
--data '{
  "name": "'${STREAMING_FUNCTION_NAME:-usd-composer}'",
  "inferenceUrl": "'${STREAMING_START_ENDPOINT:-/sign_in}'",
  "inferencePort": '${STREAMING_SERVER_PORT:-49100}',
  "health": {
        "protocol": "HTTP",
        "uri": "/v1/streaming/ready",
        "port": '${CONTROL_SERVER_PORT:-8111}',
        "timeout": "PT10S",
        "expectedStatusCode": 200
  },
  "containerImage": "'$STREAMING_CONTAINER_IMAGE'",
  "apiBodyFormat": "CUSTOM",
  "description": "'${STREAMING_FUNCTION_NAME:-usd-composer}'",
  "functionType": "STREAMING",
  "containerEnvironment": [
        {"key": "NVDA_KIT_NUCLEUS", "value": "'$NUCLEUS_SERVER'"},
        {"key": "OMNI_JWT_ENABLED", "value": "1"},
        {"key": "VECTOR_OTEL_ACTIVE", "value": "TRUE"},
        {"key": "NVDA_KIT_ARGS", "value": "--/app/livestream/nvcf/sessionResumeTimeoutSeconds=300"}
  ],
  "telemetries": {
        "logsTelemetryId": "'$TELEMETRY_ID'",
        "metricsTelemetryId": "'$TELEMETRY_ID'"
  }
}'

Confirm Logs within Azure Monitor

In the Azure Portal, open the Log Analytics Workspace created for Application Insights. Select the resource, click View Logs, switch to KQL mode, and run the sample query below.

AppTraces
| where Properties.function_id == "xxxxxxxxxxxx"