Integrate Vector Log Processor with Kit Container#
Configure Vector with a Custom Entrypoint Script#
Create a entrypoint_vector_dev.sh
in your working directory:
touch entrypoint_vector_dev.sh
Script Components Overview#
The custom entrypoint script orchestrates Vector configuration and application startup. It handles multiple critical functions:
Environment Setup:
User environment configuration for Kit application
Debug logging for troubleshooting
Environment variable validation
Vector OTel Processing Control:
Checks
VECTOR_OTEL_ACTIVE
flag to enable/disable Vector processingFalls back to standard Kit execution if Vector is disabled
Configuration Management:
Supports both custom configuration
VECTOR_CONF_B64
and static configurationDecodes base64-encoded Vector configuration
Replaces OTel endpoint placeholders with actual values
Creates log file directories
/tmp/kit_structured_logs.log
Validation & Testing:
Vector configuration syntax validation
OTel endpoint connectivity testing
Network connectivity verification to env var
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
Process Orchestration:
Starts Vector log processor in background
Launches Kit application with log capture
Pipes Kit output to log files for Vector processing
Add the Script Content#
Copy the following in the entrypoint script:
nano entrypoint_vector_dev.sh
#!/bin/bash
echo "[entrypoint_vector_dev.sh] Starting container..."
# Set required environment variables for Kit
export USER="ubuntu"
export LOGNAME="ubuntu"
# Check if Vector OTEL processing is enabled
if [ "$VECTOR_OTEL_ACTIVE" = "TRUE" ]; then
echo "[Vector] Vector OTEL processing is ENABLED (VECTOR_OTEL_ACTIVE=TRUE)"
# Create log files if they don't exist
echo "[Vector] Setting up log files..."
touch /tmp/kit_structured_logs.log
chmod 666 /tmp/kit_structured_logs.log
# Validate OTEL endpoint
if [ -z "$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT" ]; then
echo "[Vector] ERROR: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT is not set!"
exit 1
fi
if [[ ! "$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT" =~ ^https?:// ]]; then
echo "[Vector] ERROR: Invalid OTEL endpoint format. Must start with http:// or https://"
exit 1
fi
echo "[Vector] Using OTEL endpoint: $OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"
# Determine which Vector configuration to use
if [ ! -z "$VECTOR_CONF_B64" ]; then
echo "[Vector] Custom Vector configuration provided via VECTOR_CONF_B64"
echo "[Vector] Decoding and using customer-provided configuration..."
# Decode Vector config
echo "$VECTOR_CONF_B64" | base64 -d > /tmp/vector_raw.toml
# Replace OTEL endpoint
sed "s|PLACEHOLDER_OTEL_ENDPOINT|$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT|g" /tmp/vector_raw.toml > /tmp/vector.toml
echo "[Vector] Using CUSTOM Vector configuration (from VECTOR_CONF_B64)"
else
echo "[Vector] No custom configuration provided. Using static/default Vector configuration..."
# Copy static configuration and replace OTEL endpoint
cp /opt/vector/static_config.toml /tmp/vector_raw.toml
sed "s|PLACEHOLDER_OTEL_ENDPOINT|$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT|g" /tmp/vector_raw.toml > /tmp/vector.toml
echo "[Vector] Using STATIC Vector configuration (from /opt/vector/static_config.toml)"
fi
# Show the first few lines of the config for debug
echo "[Vector] First 10 lines of /tmp/vector.toml:" && head -n 10 /tmp/vector.toml
# Validate Vector config
echo "[Vector] Verifying Vector configuration..."
if [ -x "/opt/vector/bin/vector" ]; then
/opt/vector/bin/vector validate /tmp/vector.toml
if [ $? -ne 0 ]; then
echo "[Vector] ERROR: Vector configuration validation failed!"
exit 1
fi
fi
echo "[Vector] Starting Kit with real-time log forwarding..."
# Start Vector reading from the log file
/opt/vector/bin/vector --config /tmp/vector.toml &
VECTOR_PID=$!
# Start Kit and pipe its output to the log file
stdbuf -oL /entrypoint.sh 2>&1 | stdbuf -oL tee -a /tmp/kit_structured_logs.log &
PIPELINE_PID=$!
# Wait for the pipeline (which includes Kit) to complete
wait $PIPELINE_PID
echo "[Vector] Kit pipeline has ended"
echo "[Vector] Cleaning up Vector process..."
# Kill Vector
kill -TERM $VECTOR_PID 2>/dev/null || true
echo "[Vector] Container exiting"
exit 0
else
echo "[Vector] Vector OTEL processing is DISABLED (VECTOR_OTEL_ACTIVE=FALSE or not set)"
echo "[Vector] Running Kit without log processing."
exec /entrypoint.sh
fi
Vector OTel Transform#
The Vector OTel Transform is responsible for converting the Kit application’s plaintext logs into OpenTelemetry (OTel) format required by NVCF. This transformation ensures your logs are properly structured and compliant with OTel standards.
Vector follows the Vector Remap Language (VRL) to interact with the log processing. You are encouraged to extend Vector’s capability as required.
Transformation Process#
Input: Kit application plaintext logs (stdout/stderr)
Processing: Vector Remap Language (VRL) transformation rules
Output: OpenTelemetry-compliant JSON logs
For the purpose of this document, we are using verified VRL syntax to work appropriately with the log processing pipeline.
The vector.toml
file being created as part of this step, will serve as the static configuration file. This step is provided as a reference to how the VRL works. This is a required step as part of the configuration.
To pass a custom vector configuration, use the VECTOR_CONF_B64
environment variable.
Create the Vector Configuration File#
touch vector.toml
nano vector.toml
Copy the Vector Configuration#
[sources.kit_logs]
type = "file"
include = ["/tmp/kit_structured_logs.log"]
read_from = "beginning"
max_line_bytes = 1024
ignore_older_secs = 86400 # Ignore logs older than 24 hours
remove_after_secs = 604800 # Remove processed files after 7 days
[transforms.otel_transforms]
type = "remap"
inputs = ["kit_logs"]
source = '''
# Extract log level from message
level = "INFO"
message = to_string(.message) ?? ""
if message != "" {
if match(message, r'\[Error\]') {
level = "ERROR"
} else if match(message, r'\[Warning\]') {
level = "WARN"
} else if match(message, r'\[Debug\]') {
level = "DEBUG"
}
}
# Set severity number based on level
severity_number = 9
if level == "ERROR" {
severity_number = 17
} else if level == "WARN" {
severity_number = 13
} else if level == "DEBUG" {
severity_number = 5
}
.resourceLogs = [{
"resource": {"attributes": [{
"key": "service.name",
"value": {"stringValue": "kit-vector-app"}
}]
},
"scopeLogs": [{
"scope": {
"name": "kitvector.log",
"version": "1.0.0"
},
"logRecords": [{
"timeUnixNano": to_string(to_unix_timestamp(now(), unit:
"nanoseconds")),
"body": {"stringValue": .message},
"severityText": level,
"severityNumber": severity_number
}]
}]
}]
'''
[sinks.otel_collector]
type = "http"
inputs = ["otel_transforms"]
uri = "${OTEL_EXPORTER_OTLP_LOGS_ENDPOINT}"
encoding.codec = "json"
method = "post"
request.headers.Content-Type = "application/json"
framing.method = "newline_delimited"
batch.max_events = 1
batch.timeout_secs = 1
request.retry_attempts = 3
request.retry_initial_backoff_secs = 1
request.retry_max_duration_secs = 10
[sinks.console_debug]
type = "console"
inputs = ["otel_transforms"]
encoding.codec = "json"
encoding.timestamp_format = "rfc3339"
This configuration allows kit logs to be read from thestdout/stderr redirected log file /tmp/kit_structured_logs.log
.
The OTel transformed logs are directed to the NVCF’s OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
& to the console. As a result, you can observe the kit application logs on the NVCF UI and your observability backend.