Omniverse Lidar Extension

Introduction

The Lidar extension consists of RtxSensor based Lidar simulation plugins (currently only one), several plugins and different Omnigraph Nodes. The simulation plugin is a generic lidar model, which can be configured via a sensor profile json file.

LidarPointCloud

The Lidar extension uses the GenericModelOutput as output format. See the documentation of the common extension for more information on the GenericModelOutput.

LidarProfile

The lidar profile parameters affect the behavior of the lidar sensor. It consists of mandatory basic parameters and optional ones. Examples are the Example_Rotary and Example_Solid_State profiles.

Basic Parameters

  • scanType: The type of the lidar sensor (rotary for rotating and solidState for non-moving sensors)

  • nearRangeM/farRangeM: The min/max range of the lidar sensor in meters

  • rangeResolutionM: The resolution of the range in meters

  • rangeAccuracyM: The accuracy of the range in meters

  • wavelengthNm: The wavelength of the lidar sensor in nanometers

  • maxReturns: Number of max returns per emitter (e.g., 2 for dual return mode)

  • reportRateBaseHz: Number of times the firing pattern is applied per second, this is defined as a “tick”

  • numberOfEmitters: Number of emitters in the lidar sensor per scan

  • scanRateBaseHz: Number of full point cloud scans per second

  • numLines (Solid State): Number of rows per scan

  • numRaysPerLine (Solid State): Number of rays per line/row, array with numLines elements

  • Firing Pattern Parameter (see Emitter State Parameters for detailed information):

    • emitters: Array of emitters if emitterStateCount is not defined

    • emitterStateCount: Number of states per emitter

    • emitterStates: Array of states for each emitter

Optional Parameters

  • avgPowerW: Average power of one lidar emitter pulse in watts (default: 0.002)

  • pulseTimeNs: Pulse time of one lidar emitter pulse in nanoseconds (default: 6)

  • validStartAzimuthDeg: Start azimuth of the valid range in degrees (default: 0)

  • validEndAzimuthDeg: End azimuth of the valid range in degrees (default: 360)

  • minDistBetweenEchos: Minimum distance between two echoes in meters (default: min(nearRangeM,0.4))

  • rangeOffset: Range offset in meters (default: 0) – useful for not hitting the prop

  • numberOfChannels: Number of channels/detectors in the lidar sensor (default: numberOfEmitters)

  • minReflectance: Minimum reflectance expected to see at minReflectanceRangeM (default: 0.1) – not the overall min reflectance

  • minReflectanceRange: Maximum range at which the minimum reflectance is expected to generate a valid return (default: 200)

  • intensityProcessing: Processing to get final intensity (supported right now: “normalization” and “correction”)

  • rayType: Type of ray in geometrical sense (e.g., “IDEALIZED”, “GAUSSIAN_BEAM”)

  • Detector Parameters:

    • effectiveApertureSize: Effective aperture size of the detector in meters (default: 0.01)

    • quantumEfficiency: Quantum efficiency of the detector (default: 0.9)

    • pixelPitch: Pixel pitch of the detector (default: 40.0)

    • calibrationGain: Calibration gain of the detector (default: 40.0)

    • bitDepthResolution: Bit depth resolution of the detector (default: 65536)

  • intensityMapping:

    • intensityMappingType: Type of intensity mapping (e.g., “LINEAR”, “NONLINEAR” – intensityMappingDecoding/Encoding)

    • intensityMappingDecoding: Decoding function values for intensity mapping

    • intensityScalePercent: Intensity scale in percent (default: 255)

  • Range Parameter: For setting a per emitter range (nearRangeM,farRangeM will be used as default)

    • rangeCount: Number of ranges in array

    • ranges: Array of ranges for each emitter e.g., [{“min”: 0, “max”: 100}, {“min”: 3, “max”: 200}]

  • Beam Parameter: Parameter of laser beam

    • beamWaistHorM: Horizontal beam waist in meters (default: 0.002)

    • beamWaistVerM: Vertical beam waist in meters (default: 0.002)

    • divergenceHorDeg: Horizontal divergence of the laser beam in degrees (default: calculated from beam waist)

    • divergenceVerDeg: Vertical divergence of the laser beam in degrees (default: calculated from beam waist)

    • focusDist: Focus distance of the laser beam in meters (default: 0.15)

    • Msquared: M squared value of the laser beam (default: 1.4)

Emitter State Parameters

The firing pattern, potentially, consists of multiple emitter states, whereas one emitter state is a full firing of all emitters. The minimum emitter parameters which have to be set are the following:

  • azimuthDeg: Azimuth of the emitter in degrees relative to the tick azimuth (see reportRateBaseHz)

  • elevationDeg: Elevation of the emitter in degrees

  • fireTimeNs: Time of the emitter firing in nanoseconds relative to the tick time (see reportRateBaseHz), should be inside the tick delta time

Additionally, the following optional parameters can be set:

  • channelId: Channel id of the emitter, to map specific emitter to channels (default: emitterId)

  • rangeId: Range id of the emitter, to map specific emitter to specified ranges (default: 0)

  • bank: Bank id of the emitter, to map specific emitter to banks/lines for solid state lidar (default: 0)

  • vertOffsetM: Vertical offset of the emitter in meters (default: 0)

  • horOffsetM: Horizontal offset of the emitter in meters (default: 0)

  • distanceCorrectionM: Distance correction of the emitter in meters (default: 0)

  • reportRateDiv: Report rate divisor of the emitter, to reduce the report rate of the emitter (default: 1)

  • focalSlope: Focal slope of the emitter, to correct the focal point of the emitter (default: 0)

  • focalDistM: Focal distance of the emitter in meters, to correct the focal point of the emitter (default: 0)

LidarPCConverter

The LidarPCConverter converts the RTXSensor Lidar data stream to an easily usable point cloud (GenericModelOutput format). The interface is defined in (include/omni/sensors/lidar/ILidarPCConverter.h).

Users can create objects that implement this interface by acquiring the ILidarPCConverterFactory (include/omni/sensors/lidar/ILidarPCConverterFactory.h) carbonite interface and using the createInstance() method.

Instantiation

To instantiate the LidarPCConverter the user needs to acquire the carbonite ILidarPCConverterFactory interface, and use the create instance method:

ILidarPCConverterPtr m_pcConverter = carb::getFramework()->acquireInterface<omni::sensors::lidar::ILidarPCConverterFactory>()->createInstance();

Initialization

The user has to initialize the converter, with the corresponding converter cfg (omni::sensors::lidar::LidarPCConverterCfg) like this:

m_pcConverter->init(cfg);

Conversion

The conversion of the buffer differs with the conversion buffer type (PACKETS, GENERIC_FILE (HDF5) or live sensor buffer). In the live sensor buffer the user calls the following function

m_pcConverter->convertBuffer(sensorBuffer, dataSize, numPoints, scanComplete, cudaStream);

Next he can get the point cloud:

LidarPointCloud pc = m_pcConverter->getPointCloud(cudaStream);

The user has to keep in mind, that in live CPU mode the getPointCloud function has to be called in a cudaLaunchHostFunc with the appropriate cudaStream, as every work is scheduled asynchronously.

LidarCorePlugin

Introduction

The Lidar core plugin is a generic Lidar simulation utilizing RtxSensor. It can be configured through sensor config file (json format) and it supports rotary as well as solid state lidars.

Plugin Name

omni.sensors.nv.lidar.lidar_core.plugin

Parameters & Attributes

The attributes are specified through a sensor config JSON file, which are placed in “data/sensors/lidar”. The desired sensor profile is set via the USDA attribute “custom token rtxsensor:modelConfig” (see Example in USDA for an example). This JSON file can be read with the ProfileReader utility plugin.

Available Parameterizations

The following parameterization to support specific Lidar already exists:

Lidar Type

Params File name

Model Config Entry

Variant

Generic Rotary

Example_Rotary.json

“Example_Rotary”

Generic Solid State

Example_Solid_State.json

“Example_Solid_State”

Atmospheric Modeling

The Lidar model includes a mie scattering based atmospheric simulation model, which currently supports rain. It is planned to add fog, haze and snow at a later time.

Besides the wavelength dependency, the rain model has one basic parameter: the rain rate in mm/h (configurable as a carb::setting: “/app/sensors/nv/atmospherics/rainRate”). For instance, a rain rate of 0.25mm/h is light rain, whereas a rain rate of 25mm/h corresponds to a very heavy rain. Every other parameter for the mie scattering (e.g., rain drop size distribution) is derived from the rain rate.

Additionally, the rain model provides the possibility to simulate rain drop hits as false positive. The amount of false positives can be set via a rain drop hit threshold (configurable as a carb::setting: “/app/sensors/nv/atmospherics/rainDropHitThresh”). This threshold is related to the mie back scattering efficiency parameter and a distance dependent rain drop hit probability distribution.

Both parameter can be set via command line are dynamically via the script editor, as both parameter are defined as a carb::setting. In example you can start the simulation with a rain rate of 25mm/h if you’re adding “–/app/sensors/nv/atmospherics/rainRate=25.” to the startup command of kit. It is important to activate the material framework to get accurate readings from wet roads (point drops) and false positives due to back scattering (under the ground).

Omnigraph Nodes

One example of the current nodes in the Lidar extension is the TranscoderLidar. A dumping transcoder, which encodes the Lidar data into vendor specific udp packets and dumps them into a bin-file.

TranscoderLidar

Introduction

The Transcoder node encodes the Lidar data stream into vendor specific packets and records them via binary file writing.

Omnigraph Node Name

omni.sensors.nv.lidar.TranscoderLidar

Parameters & Attributes

Parameter Name

Description

Value Type

Value Range

Default Value

Example

sensorProfileName

Filename of sensor profile (excluding directory and .json)

string

“GENERIC”

returnType

Indicates the desired return (first,last,dual,etc.,)

int

0

runMode

CPU = 0 or GPU = 1

int

0

dumpPackets

Dump packets as file

bool

false

decoderPath

Optional path to ndas decoder

string

“”

fileName

Filename of the dumped bin file

string

“”

Python Bindings and Scripts

The Lidar extension offers a variety of python bindings and scripts for the Lidar. These can be used for easy post-processing of the simulated Lidar point cloud.

Python Bindings

The python bindings make it possible to use the Lidar utilities and specific classes in python code. For instance, the user can import the Lidar python bindings in the following way:

import omni.sensors.nv.lidar.bindings._lidar as lidar

Furthermore, there exist the LidarUtilities script, containing some helper functions for ease of use of the Lidar bindings. The script can be imported, e.g., the following way:

import omni.sensors.nv.lidar.scripts.LidarUtilities as lutil

There are binding definitions for all basic classes and structs of the Lidar and lidar Tools extension. See the following (incomplete) list of the most basic classes:

  • LidarProfile

Furthermore, there are bindings for the following utilities:

  • LidarPCConverter

The binding exists for the LidarPCConverter, which accumulates and converts Lidar sensor buffers to the LidarPointCloud. There is a mode to accumulate and convert udp packets to a Lidar point cloud, but not every sensor is supported. To get a LidarPCConverter object the user can call the following helper function of the LidarUtilities script

cfg = lutil.PCGeneratorConfig(bin_file) # or (bin_file,"",sensor_name) for reading a HDF5 file (only supported in linux for now)
converter = lutil.get_pc_converter(cfg)

For more options of the PCGeneratorConfig see the LidarUtilities script.

Currently, only host side conversion of the buffer is supported, but GPU accelerated conversion will be added in the near future. An example conversion is as follows

# convert buffer
# File is specified in the cfg
converter.convertBuffer()
# or this for a GenericModelOutput binary buffer
# converter.convertBuffer(buffer,data_size, num_points, scan_complete)
pc = converter.getPointCloud()