RTX Sensor Visual Materials

There are 21 sensor materials that are rendered in the visual spectrum, and more can not be added at this time. Their properties are stored in json files by the same name, located in the ./data/material_files/ folder.

Index	Sensor Material Type
0	Default
1	AsphaltStandard
2	AsphaltWeathered
3	VegetationGrass
4	WaterStandard
5	GlassStandard
6	FiberGlass
7	MetalAlloy
8	MetalAluminum
9	MetalAluminumOxidized
10	PlasticStandard
11	RetroMarkings
12	RetroSign
13	RubberStandard
14	SoilClay
15	ConcreteRough
16	ConcreteSmooth
17	OakTreeBark
18	FabricStandard
19	PlexiGlassStandard
20	MetalSilver
31	INVALID

How to use sensor materials

To use sensor materials, Isaac Sim must know how to map indices (seen in the Material ID return) to the sensor material type in the table above. This is done by setting the following carb settings in ./exts/omni.isaac.sensor/config/extension.toml [settings] section.

rtx.materialDb.rtSensorNameToIdMap="DefaultMaterial:0;AsphaltStandardMaterial:1;AsphaltWeatheredMaterial:2;VegetationGrassMaterial:3;WaterStandardMaterial:4;GlassStandardMaterial:5;FiberGlassMaterial:6;MetalAlloyMaterial:7;MetalAluminumMaterial:8;MetalAluminumOxidizedMaterial:9;PlasticStandardMaterial:10;RetroMarkingsMaterial:11;RetroSignMaterial:12;RubberStandardMaterial:13;SoilClayMaterial:14;ConcreteRoughMaterial:15;ConcreteSmoothMaterial:16;OakTreeBarkMaterial:17;FabricStandardMaterial:18;PlexiGlassStandardMaterial:19;MetalSilverMaterial:20"

With the rtx.materialDb.rtSensorNameToIdMap set, you must edit a csv file to map exact material name tokens to sensor material types. This csv file is located at ./kit/rendering-data/runtime/RtxSensorMaterialMap.csv.

The RtxSensorMaterialMap.csv file contains a material prim partial names to sensor material type pairs. The ones that come with Isaac Sim by default can be deleted as they may clash with names you wish to set. There is only one csv file. It controls the material mapping for all of the content. It is read at Isaac Sim startup and any changes to it require a restart to show up.

As an example, consider this scene:

The /Root/SM_floor29/SM_floor02/SM_floor02 prim, has a material prim assigned to it whose path is /Root/SM_floor29/Looks/MI_Floor_02b. If you want to add an entry to the csv file so that the SM_floor02 prim looks like rough concrete to the rtx sensors, you would add the entry:

mi_floor_02b,ConcreteRoughMaterial

Note that in the csv mapping file, the first token after the first appearance of /Looks/ in the material prim name attached to the mesh is used, and it must always be lowercase in the csv file, no matter what the case is on the stage. Also note how the word Material is concatenated onto the sensor material type from the table above.

RTX Sensor Non-Visual Materials

The RTX Sensor extensions now include support for rendering non-visual materials. To enable non-visual material rendering, you will need to run Isaac Sim from a terminal with additional settings.

From a Linux terminal:

$ ./isaac-sim.sh --/rtx/materialDb/nonVisualMaterialCSV/enabled=false

From a Windows command prompt:

> .\isaac-sim.bat --/rtx/materialDb/nonVisualMaterialCSV/enabled=false

You can also enable the setting when running Isaac Sim with Python using:

import carb
carb.settings.get_settings().set_bool("/rtx/materialDb/nonVisualMaterialCSV/enabled", False)

To run the scene, open Isaac Sim using two additional settings:

From a Linux terminal:

$ ./isaac-sim.sh \
--/rtx/materialDb/nonVisualMaterialCSV/enabled=false \
--/app/rtxSensorSample/contentMaterialTest/enabled=true \
--/rtx/rtxsensor/coordinateFrameQuaternion="0.5,-0.5,-0.5,-0.5"

From a Windows command prompt:

> .\isaac-sim.bat ^
--/rtx/materialDb/nonVisualMaterialCSV/enabled=false ^
--/app/rtxSensorSample/contentMaterialTest/enabled=true ^
--/rtx/rtxsensor/coordinateFrameQuaternion="0.5,-0.5,-0.5,-0.5"

Note

By default, /rtx/rtxsensor/coordinateFrameQuaternion is set to "1.0,0.0,0.0,0.0". This specific example was built using the legacy quaternion, hence the changed setting.

Use the content browser to navigate to Isaac/Samples/Examples/NonVisualMaterials/RtxSensorSample.usda, then open the scene. Select the LegacyRtxSensorSample camera view. You should see a view like the following:

Now select the RtxSensorSample camera view. You should see a view like the following:

Note the change in colors and reflections due to non-visual material attributes defined in the .usda file. An example portion of the file is provided below:

def Material "GreenAlbedo"
{
    # ... additional attributes
    custom string inputs:nonvisual:attributes = "emissive"
    custom string inputs:nonvisual:coating = "none"
    custom string inputs:nonvisual:coating = "rubber"
    # ... additional attributes
}

The full set of non-visual material attributes is described in the Omniverse Non-Visual Materials Extension documentation. Attributes may be added to materials from the UI by right-clicking the material in the Stage window, then selecting Add > Attribute. This will open a new window like the one below, enabling you to specify custom non-visual attributes.

After adding the new attribute, it will appear in the material’s properties, at which point it can be populated:

Debugging

The carb parameter:

[settings]
rtx.materialDb.rtSensorMaterialLogs=true

can help. If set to true it will output a list of all the materials in the scene that are NOT mapped to a sensor material. This list outputs to the terminal and the log at Isaac Sim startup.