Reusable Light Panel

Gestures are a powerful way to allow you to create scenes with interactive objects. In this hands-on tutorial you will create a light panel where users can alter the size and intensity of the light by dragging its edges.

Learning Objectives

In this tutorial, you will learn how to:

• Extend the Manipulator class to create a light panel with a basic model based UI.

• Use Gestures to customize and extend an object’s UI.

Prerequisites

It is recommended to understand the concepts in the following tutorials before proceeding:

• How to make an extension that spawns primitives

• Learn how to add extensions to the Omniverse Extension Manager

Note

You must use Omniverse Code version 2022.1.2 or newer.

Step 1: Download the Starter Project

Clone the tutorial-start branch of the kit-extension-sample-ui-scene repository to get the assets for this tutorial: NVIDIA-Omniverse/kit-extension-sample-ui-scene

This tutorial will focus on the light_manipulator.py file found in the /exts/omni.example.ui_scene.light_manipulator/omni/example/ui_scene/light_manipulator directory. To learn more about the other files in the repository check out the How to make an extension that spawns primitives.

Step 2: Familiarize Yourself with on_build() of the LightManipulator Class

This tutorial will be focusing on the LightManipulator class, which is created by extending the Manipulator class. By doing this, it enables you to create an interactive rectangular light, a LightPanel, as shown in the image below:

LightManipulator overrides on_build() from Manipulator and is where you will be adding the custom code. It should initially look like this:

def on_build(self):
    """Called when the model is changed and rebuilds the whole slider"""
    model = self.model
    if not model:
        return

    # if we don't have selection then just return
    prim_path_item = model.prim_path
    prim_path = prim_path_item.value if prim_path_item else None
    if not prim_path:
        return

    #TODO: Add Step 3

    self.__root_xf = sc.Transform(model.get_as_floats(model.transform))
    with self.__root_xf:
        self._x_xform = sc.Transform()
        with self._x_xform:
            self._shape_xform = sc.Transform()
            # Build the shape's transform
            self._build_shape()
            with self._shape_xform:
                #TODO: Add Step 4.1

                #TODO: Add Step 4.2

                #TODO: Add Step 5.1

                #TODO: Add Step 6.1

                #TODO: Add Step 6.2

                #TODO: Add Step 7.1

                #TODO: Add Step 7.2
                pass

When a Manipulator needs updating on_build() is called and uses a Model, as defined in light_model.py.

This self.model has a number of useful functions such as getting and setting the height, width and intensity of the light. get_as_floats() and set_floats() allow you to retrieve or alter all of these values all at once.

If self.model is not set the function ends since there isn’t anything to manipulate. Similarly, there must also be a RectLight available in prim_path.

Step 3: Setting Line Style of the LightPanel

To show the user the LightPanel is resizable, and which edge of its rectangle is being hovered over, set the style and thickness of its edges:

Copy the following code under the # Add Step 3 comment in on_build(). This will not render anything in Omniverse yet, but will provide a setup for later.

# Style settings, as kwargs
thickness = 1
hover_thickness = 3
color = cl.yellow
shape_style = {"thickness": thickness, "color": color}

def set_thickness(sender, shapes, thickness):
    for shape in shapes:
        shape.thickness = thickness

Here you created set_thickness() for the HoverGesture to use as a callback for adjusting the size of the edge that’s being hovered over.

CHALLENGE (optional): Want to make the style your own? Try changing the thickness, hover_thickness and color variables to your liking. It may be worth revisiting this code and adjusting it over the course of this tutorial.

Step 4: Building the Manipulator Transform

To make it easier for the user to scale the LightPanel, give it a Transform.

A Translation Transform is created with the following code:

Note

Please do not copy this code

self.__root_xf = sc.Transform(model.get_as_floats(model.transform))
with self.__root_xf:
    self._x_xform = sc.Transform()
    with self._x_xform:
        self._shape_xform = sc.Transform()
        # Build the shape's transform
        self._build_shape()

This code will start from the top down, store the root Transform of the Model in self.__root_xf. Then, the code creates self._x_xform, a Transform for translation and it also creates a Transform for the shape, self._shape_xform:

Step 4.1: Set the Scale

Add this code to # Add Step 4.1 in on_build():

def _build_shape(self):
    if not self.model:
        return
    if self.model.width and self.model.height and self.model.intensity:
        x = self.model.get_as_floats(self.model.width)
        y = self.model.get_as_floats(self.model.height)
        # this INTENSITY_SCALE is too make the transform a reasonable length with large intensity number
        z = self.model.get_as_floats(self.model.intensity) / INTENSITY_SCALE
        self._shape_xform.transform = [x, 0, 0, 0, 0, y, 0, 0, 0, 0, z, 0, 0, 0, 0, 1]

Finally, _build_shape() sets the scale on the _shape_xform using the vector [x, 0, 0, 0, 0, y, 0, 0, 0, 0, z, 0, 0, 0, 0, 1] where x is width, y is the height, and z is the light intensity. This list is being used as a scale matrix.

Step 4.2: Build the Geometry

Now that the transform has been created you need to render the rectangle’s geometry.

Copy the below code block under the # Add Step 4.2 comment in on_build(). Be sure to check the alignment of the python code and feel free to view the completed function as a hint.

# Build the shape geometry as unit-sized
h = 0.5
z = -1.0
# the rectangle
shape1 = sc.Line((-h, h, 0), (h, h, 0), **shape_style)
shape2 = sc.Line((-h, -h, 0), (h, -h, 0), **shape_style)
shape3 = sc.Line((h, h, 0), (h, -h, 0), **shape_style)
shape4 = sc.Line((-h, h, 0), (-h, -h, 0), **shape_style)

There should now be a rectangle viewable in Omniverse.

Step 5: Handling Gestures

Gestures handle callback data from user input. There are many kinds of Gestures in Omniverse, listed here.

This tutorial focuses on the Hover and Drag gestures.

Step 5.1: Add the HoverGesture for the RectLight

Copy the below code block under the # Add Step 5.1 comment on_build(). Again, make sure that this code aligns with the comment.

# add gesture to the lines of the rectangle to update width or height of the light
vertical_hover_gesture = sc.HoverGesture(
    on_began_fn=lambda sender: set_thickness(sender, [shape1, shape2], hover_thickness),
    on_ended_fn=lambda sender: set_thickness(sender, [shape1, shape2], thickness),
)
shape1.gestures = [_DragGesture(self, [1], [1]), vertical_hover_gesture]
shape2.gestures = [_DragGesture(self, [1], [-1]), vertical_hover_gesture]

horizontal_hover_gesture = sc.HoverGesture(
    on_began_fn=lambda sender: set_thickness(sender, [shape3, shape4], hover_thickness),
    on_ended_fn=lambda sender: set_thickness(sender, [shape3, shape4], thickness),
)
shape3.gestures = [_DragGesture(self, [0], [1]), horizontal_hover_gesture]
shape4.gestures = [_DragGesture(self, [0], [-1]), horizontal_hover_gesture]

To change the thickness of the edges of your RectLight while the user hovers over them, set the on_began_fn() (beginning of the hover) and on_ended_fn() (ending of the hover) to use set_thickness() defined in Step 3: Setting Line Style of the LightPanel

Since the rectangle will grow and shrink about its center, you will highlight both of the affected edges to reflect this mirrored behavior. For instance, if you’re changing the vertical height of the rectangle, both the top and bottom edges will be highlighted like so:

Step 5.2: Analyzing the DragGesture

The DragGesture will do a number of different things to your light panel depending on which edge is dragged. You’ll extend the base DragGesture class into _DragGesture.

Take a look at a key part of the on_began() function:

Note

Please do not copy this code

if 0 in self.orientations:
    self.width_item = self.model.width
    self._manipulator.model.set_item_value(self.width_item, self.model.get_as_floats(self.width_item))
if 1 in self.orientations:
    self.height_item = self.model.height
    self._manipulator.model.set_item_value(self.height_item, self.model.get_as_floats(self.height_item))
if 2 in self.orientations or self.is_global:
    self.intensity_item = self.model.intensity
    self._manipulator.model.set_item_value(self.intensity_item, self.model.get_as_floats(self.intensity_item))

When defining the Gesture, you pass in an Orientation to let it know whether it’s for width, height, or intensity. Then, you will alter the respective variable. Finally, to build the Gesture, call it like so:

_DragGesture(manipulator, orientation, flag)

Where flag indicates the negative or positive edge, e.g., the left horizontal edge v. the right horizontal edge.

Step 6: Adding Intensity

While you have learned how to manipulate the light’s height and width, it would be useful if users could also define the intensity of the light using the DragGesture. To make it clear to the user that they are using the panel’s local z-axis to modify its intensity, you can use arrows.

Step 6.1: Create Custom Arrows

Add this code to # Add Step 6.1 in on_build()

# create z-axis to indicate the intensity
z1 = sc.Line((h, h, 0), (h, h, z), **shape_style)
z2 = sc.Line((-h, -h, 0), (-h, -h, z), **shape_style)
z3 = sc.Line((h, -h, 0), (h, -h, z), **shape_style)
z4 = sc.Line((-h, h, 0), (-h, h, z), **shape_style)

def make_arrow(translate):
    vert_count = len(ARROW_VI)
    with sc.Transform(
        transform=sc.Matrix44.get_translation_matrix(translate[0], translate[1], translate[2])
        * sc.Matrix44.get_rotation_matrix(0, -180, 0, True)
    ):
        return sc.PolygonMesh(ARROW_P, [color] * vert_count, ARROW_VC, ARROW_VI, visible=False)

# arrows on the z-axis
arrow_1 = make_arrow((h, h, z))
arrow_2 = make_arrow((-h, -h, z))
arrow_3 = make_arrow((h, -h, z))
arrow_4 = make_arrow((-h, h, z))

# the line underneath the arrow which is where the gesture applies
z1_arrow = sc.Line((h, h, z), (h, h, z - ARROW_HEIGHT), **shape_style)
z2_arrow = sc.Line((-h, -h, z), (-h, -h, z - ARROW_HEIGHT), **shape_style)
z3_arrow = sc.Line((h, -h, z), (h, -h, z - ARROW_HEIGHT), **shape_style)
z4_arrow = sc.Line((-h, h, z), (-h, h, z - ARROW_HEIGHT), **shape_style)

The arrow’s mesh has already been defined at the top of light_manipulator.py with the variables ARROW_P, ARROW_VI, and ARROW_VC. make_arrow() makes use of these variables to draw the arrows.

While this draws the arrows, you still need to add the HoverGesture to show them…

Step 6.2: Adding the HoverGesture to the Arrows

Add this code to # Add Step 6.2 in on_build()

def set_visible(sender, shapes, thickness, arrows, visible):
    set_thickness(sender, shapes, thickness)
    for arrow in arrows:
        arrow.visible = visible

thickness_group = [z1, z1_arrow, z2, z2_arrow, z3, z3_arrow, z4, z4_arrow]
visible_group = [arrow_1, arrow_2, arrow_3, arrow_4]
visible_arrow_gesture = sc.HoverGesture(
    on_began_fn=lambda sender: set_visible(sender, thickness_group, hover_thickness, visible_group, True),
    on_ended_fn=lambda sender: set_visible(sender, thickness_group, thickness, visible_group, True),
)
gestures = [_DragGesture(self, [2], [-1]), visible_arrow_gesture]
z1_arrow.gestures = gestures
z2_arrow.gestures = gestures
z3_arrow.gestures = gestures
z4_arrow.gestures = gestures

This code creates a function, set_visible(), that alters both the visibility of the arrows and the thickness of the edge lines as part of your HoverGesture. Then, adds the DragGestures in order to change the light intensity while dragging an arrow.

CHALLENGE (Optional): Right now, the arrows are always visible regardless if the cursor is hovering over them or not. Can you alter the code so that the arrows only appear when hovering? Only one line needs to be changed. Click the below for a hint.

Solution

In on_ended_fn, the visible parameter (the last argument of the function) should be set to False.

Step 7: Scaling Everything

All the critical features of the extension have been implemented so far:

• The ability to change the width and height of a light panel

• The ability to change the intensity of a light panel.

In many 3D apps, there is often a shortcut to scale all of the dimensions of an object at once. Add a few small rectangles to the corners of the LightPanel to allow users to scale width and height at the same time.

Step 7.1: Add Scaling Rectangles

Add this code to # Add Step 7.1 in on_build()

# create 4 rectangles at the corner, and add gesture to update width, height and intensity at the same time
s = 0.03

def make_corner_rect(translate):
    with sc.Transform(transform=sc.Matrix44.get_translation_matrix(translate[0], translate[1], translate[2])):
        return sc.Rectangle(s, s, color=0x0)

r1 = make_corner_rect((h - 0.5 * s, -h + 0.5 * s, 0))
r2 = make_corner_rect((h - 0.5 * s, h - 0.5 * s, 0))
r3 = make_corner_rect((-h + 0.5 * s, h - 0.5 * s, 0))
r4 = make_corner_rect((-h + 0.5 * s, -h + 0.5 * s, 0))

This defines the function make_corner_rec() to draw a rectangle.

Step 7.2: Highlight All of the Edges at Once

Add this code to # Add Step 7.2 in on_build()

def set_color_and_visible(sender, shapes, thickness, arrows, visible, rects, color):
    set_visible(sender, shapes, thickness, arrows, visible)
    for rect in rects:
        rect.color = color

highlight_group = [shape1, shape2, shape3, shape4] + thickness_group
color_group = [r1, r2, r3, r4]
hight_all_gesture = sc.HoverGesture(
    on_began_fn=lambda sender: set_color_and_visible(sender, highlight_group, hover_thickness, visible_group, True, color_group, color),
    on_ended_fn=lambda sender: set_color_and_visible(sender, highlight_group, thickness, visible_group, False, color_group, 0x0),
)

r1.gestures = [_DragGesture(self, [0, 1], [1, -1]), hight_all_gesture]
r2.gestures = [_DragGesture(self, [0, 1], [1, 1]), hight_all_gesture]
r3.gestures = [_DragGesture(self, [0, 1], [-1, 1]), hight_all_gesture]
r4.gestures = [_DragGesture(self, [0, 1], [-1, -1]), hight_all_gesture]

You’ve defined set_color_and_visible() in order to highlight all the edges at once with your HoverGesture. Then, the code passes in multiple orientations ([0, 1]) to the _DragGesture to indicate you’re altering the width and height at the same time.

Step 8: Congratulations!!

Great job getting through this tutorial. Interested in improving your skills further? Please consider checking out the Reticle Extension Tutorial.