2. Manually Commanding the Robot

2.1. Learning Objectives

In this tutorial, we’ll learn how to select the motion controller target prim and drag it around manually from the GUI to move the robot’s end-effector. We’ll also learn how to switch command modes between position-only and full-pose commands.

Note that this motion controller target prim is what the behavior scripts use to command the robot, so when running any behavior you’ll see the prim moving and directing the robot from place to place.

Prerequisite

All commands are run from the cortex examples directory:

cd standalone_examples/cortex

2.2. Preliminaries

Start Cortex into its default belief-only mode with the default Franka blocks environment.

./cortex launch

See Cortex Modes: Belief, Sim, Real for other modes. Manual control can be used in all modes.

Make sure there isn’t currently an active behavior running. By default, there won’t be on startup. But if you’ve already been running a different behavior, clear it using

./cortex clear

2.3. Selecting the target prim and dragging it manually

Go to the stage browser on the upper right panel of the app; select the “Stage” tab if it isn’t already selected. Select /cortex/belief/motion_controller_target. (You may have to expand both cortex and belief to find it.) See the figure:

Stage path to motion controller target.

The motion controller target prim is a small grey cube with 1cm sides. It may not be visible behind the tip of the gripper currently.

Select the “Move” tool from the toolbar along the left edge of the app. A control gizmo will appear for moving the motion controller target. Move the target manually; you should see the robot move along with it.

2.4. Commanding the motion controller target from a script

Run the go_home.py behavior. You should see the motion controller target move to the home position and direct the robot home.

./cortex activate go_home.py

The go_home.py behavior goes idle once it reaches its home configuration. Once it’s there, you can manually control it again without explicitly clearing the behavior. Try playing around with moving the end-effector, including its orientation (select the “Rotate” tool on the left), and sending it back home.

Behavior scripts such as go_home.py use the MotionCommander API to command the robot, and that API in turn commands this motion controller target prim. The basic API takes the form:

self.context.tools.commander.set_command(
        MotionCommand(target, approach_params))

See exts/omni.isaac.cortex/omni/isaac/cortex/motion_commander.py. for details on the motion command API, and see Scripting Behaviors for details on programming behaviors.

2.5. Changing command modes

On startup, the motion commander is set to command the full pose of the end-effector. We can set the commander to follow only the position of the motion controller target instead. Run

./cortex activate set_commander_to_position_only.py

You might immediately see the arm relax into a more natural configuration while maintaining the same end-effector position. Now dragging the motion controller target will command only the end-effector’s position.

Set the commander back to tracking the full target:

./cortex activate set_commander_to_full_pose.py

Send the robot home:

./cortex activate go_home.py

Note: The sample behaviors in standalone_examples/cortex sometimes require full-pose commands and sometimes require position-only commands, depending on the script. Each of the scripts explicitly set the command mode using

tools.commander.set_target_full_pose()  # either this
tools.commander.set_target_position_only()  # or this

when the behaviors are built. If you examine the implementations of set_commander_to_full_pose.py and set_commander_to_position_only.py, you’ll find that those methods are used under the hood.