LIDAR Simulation

Simulating a lidar sensor is done primarily through the use of the Isaac Sim lidar sensor extension. This extension provides the API for creating lidar prims in a stage, setting the parameters of that lidar prim, and querying depth data from that prim.

Kit Editor Operations

Here we will cover the basic usage of the lidar extension from within the kit editor.

First, you must enable the extension by selecting Windows -> Extension Manager from the menus at the top of the Kit window. Scroll down and make sure that the lidar extension is activated. It should look like the highlighted line below.

Extension Window

If it does not, click on the check box to activate it. When active, you should be able to find the lidar sensor under Create -> Isaac -> Sensors -> Lidar from the drop down menus at the top of the Kit Window.

The lidar will execute a set of line PhysX traces every tick, so we need a physics scene available on the stage. To create a physics scene, select Physics -> Add -> Physics Scene from the menu bar. You should see “physicsScene” and “physicsMaterials” appear in the context panel on the right.

Create the lidar by selecting Create -> Isaac -> Sensors -> Lidar. Select the lidar in the context panel, then check the “drawLidarPoints” box in the details tab below. In that same tab, set the “rotationRate” to 1.0 and press the play button.

Note that you can update all of the lidar parameters on the fly while the stage is running. When the rotation rate reaches zero or less, the lidar prim will simply cast rays in all directions based on your FOV and resolution. Set the “rotationRate” to zero for now.

Lidar line traces will ignore anything that doesn’t have a collision API attached to it. Let’s make something that will interact with our lidar sensor. Right click in the viewport and select Create -> Shapes -> Cone. You should see a cone appear in your viewport where you clicked. Select the cone, and set its position to (200,0,0) in the details panel. Next, select the PhysXP Properties tab and click on the dropdown menu next to “Add Prim Component” and select “CollisionAPI”.

Lidar and Cone

For most use cases, the lidar will be attached to another prim (cars, robots, etc…). To do this, first we need an prim to which we can attach the lidar. Right click in the viewport and select Create -> Shapes -> Capsule. In the details panel, set the position of the capsule to (0,0,0). Finally, in the context panel, click and drag the lidar onto the capsule.

The Lidar Python API

The lidar python API can be used to interact with the lidar through scripts and extensions, and allows you to retrieve the depth data from the last sweep. In order to demonstrate this API we will use Kits python interpreter instead of writing our own extension in order to focus on the lidar.

First, we will need a fresh stage. You can make a new, empty stage by selecting File -> New from the menu bar at the top of kit. Next, we need to get the python interpreter. Select Window -> Script Editor from the menu bar. You will see a new tab open in the same frame as the viewport. Drag the Script Editor tab down to the bottom and dock it in a different frame. You should now see both the viewport and the Script Editor. At any time, you can select Command -> Execute in order to run your python code.

First, our imports…

1
2
3
4
import omni
from omni.isaac.lidar import _lidar
import omni.isaac.LidarSchema as LidarSchema
from pxr import Usd, UsdGeom, Sdf, Gf, PhysicsSchema

In order to use our lidar, we will need access to the LidarSchema, which is used to create the lidar prim itself, and the python bindings from omni isaac, which we will use to interact with our lidar prim. We also need some core functionality from the Pixar USD modules.

First, we need to get the stage, the lidar interface (so we can interact with the lidar), and the editor interface (so we can play and pause the stage)

1
2
3
stage = omni.usd.get_context().get_stage()
lidarInterface = _lidar.acquire_lidar_interface()
editorInterface = omni.kit.editor.get_editor_interface()

In an extension, this interface must be acquired when the extension starts. After we are done using it, we must also release it, which is must be done when the extension shuts down. See the Extension docs for details on writing and using your own kit extensions.

Now that we have the stage, we can create our lidar

1
2
3
scene = PhysicsSchema.PhysicsScene.Define(stage, Sdf.Path("/World/physicsSceneName"))
lidarPath = "/World/LidarName"
lidar = LidarSchema.Lidar.Define(stage,Sdf.Path(lidarPath))

Remember, our lidar is using PhysX for its line traces, so we first need to create a physics scene. Like the lidar, the physics scene is defined through the use of a schema. Defining a prim via schema requires a stage for the prim, and the desired path for the prim. When we execute this code, we will see a prim named “physicsSceneName” of type “PhysicsScene” and a prim named “LidarName” of type “Lidar” appear in the stage context menu.

In order to set any attribute on our lidar, the attribute must first be created for the prim.

1
2
3
4
5
6
7
8
9
lidar.CreateRotationRateAttr().Set(20.0)
lidar.CreateHorizontalFovAttr().Set(360.0)
lidar.CreateHorizontalResolutionAttr().Set(15.0)
lidar.CreateVerticalFovAttr().Set(45.0)
lidar.CreateVerticalResolutionAttr().Set(15.0)
lidar.CreateMinRangeAttr().Set(0.4)
lidar.CreateMaxRangeAttr().Set(100.0)
lidar.CreateHighLodAttr().Set(False)
lidar.CreateDrawLidarPointsAttr().Set(True)

Once created, we can set an attribute by “getting” it first

1
lidar.GetRotationRateAttr().Set(0.0)

We also need an obstacle for our lidar. For this we will create a cube, and place it next to the lidar. We also need to increase the size as by default the cube will be one unit on a side (which by default is 1 cm).

1
2
3
4
5
CubePath = "/World/CubeName"
cubeGeom = UsdGeom.Cube.Define(stage, CubePath)
cubePrim = stage.GetPrimAtPath(CubePath)
cubeGeom.AddTranslateOp().Set(Gf.Vec3f(0.0, 0.0, 200.0))
cubeGeom.CreateSizeAttr(100)

Again, this obstacle needs the physics collision API in order to interact with the lidar line traces

1
2
3
collisionAPI = PhysicsSchema.CollisionAPI.Apply(cubePrim)
collisionAPI.CreatePhysicsMaterialRel()
collisionAPI.CreateCollisionGroupRel()

The lidar needs “time” in order to sweep, so we will play and then pause the stage to populate the depth buffers in the lidar. The editor doesn’t need to be paused in order to retrieve the buffer. It can be fetched live at any time.

1
2
editorInterface.play()
editorInterface.pause()

Finally, we can pull the depth data from the c++ lidar object being wrapped by our python bindings. The depth data is formatted as uint16 so it can be consumed by the ISDK and various micro-controllers. To convert to the fraction of the max depth simply divide by 65535.0 (the max uint16).

1
2
3
4
5
6
7
8
maxDepth = lidar.GetMaxRangeAttr().Get()
depth = lidarInterface.get_depth_data(lidarPath)
zenith = lidarInterface.get_zenith_data(lidarPath)
azimuth = lidarInterface.get_azimuth_data(lidarPath)

print("depth", depth)
print("zenith", zenith)
print("azimuth", azimuth)