2. Manually Commanding the Robot¶
2.1. Learning Objectives¶
In this tutorial, we’ll learn how to select the motion controller target prim and drag it around manually from the GUI to move the robot’s end-effector. We’ll also learn how to switch command modes between position-only and full-pose commands.
Note that this motion controller target prim is what the behavior scripts use to command the robot, so when running any behavior you’ll see the prim moving and directing the robot from place to place.
Prerequisite
Completed the Overview and Fundamentals tutorial to understand the basic concepts, including how to launch Cortex.
All commands are run from the cortex examples directory:
cd standalone_examples/cortex
2.2. Preliminaries¶
Start Cortex into its default belief-only mode with the default Franka blocks environment.
./cortex launch
See Cortex Modes: Belief, Sim, Real for other modes. Manual control can be used in all modes.
Make sure there isn’t currently an active behavior running. By default, there won’t be on startup. But if you’ve already been running a different behavior, clear it using
./cortex clear
2.3. Selecting the target prim and dragging it manually¶
Go to the stage browser on the upper right panel of the app; select the “Stage” tab if it isn’t already
selected. Select /cortex/belief/motion_controller_target
. (You may have to expand both cortex
and belief
to find it.) See the figure:

The motion controller target prim is a small grey cube with 1cm sides. It may not be visible behind the tip of the gripper currently.
Select the “Move” tool from the toolbar along the left edge of the app. A control gizmo will appear for moving the motion controller target. Move the target manually; you should see the robot move along with it.
2.4. Commanding the motion controller target from a script¶
Run the go_home.py
behavior. You should see the motion controller target move to the home
position and direct the robot home.
./cortex activate go_home.py
The go_home.py
behavior goes idle once it reaches its home configuration. Once it’s there, you
can manually control it again without explicitly clearing the behavior. Try playing around with
moving the end-effector, including its orientation (select the “Rotate” tool on the left), and
sending it back home.
Behavior scripts such as go_home.py
use the MotionCommander
API to command the robot, and
that API in turn commands this motion controller target prim. The basic API takes the form:
self.context.tools.commander.set_command(
MotionCommand(target, approach_params))
See exts/omni.isaac.cortex/omni/isaac/cortex/motion_commander.py
. for details on the motion
command API, and see Scripting Behaviors for details on programming
behaviors.
2.5. Changing command modes¶
On startup, the motion commander is set to command the full pose of the end-effector. We can set the commander to follow only the position of the motion controller target instead. Run
./cortex activate set_commander_to_position_only.py
You might immediately see the arm relax into a more natural configuration while maintaining the same end-effector position. Now dragging the motion controller target will command only the end-effector’s position.
Set the commander back to tracking the full target:
./cortex activate set_commander_to_full_pose.py
Send the robot home:
./cortex activate go_home.py
Note: The sample behaviors in standalone_examples/cortex
sometimes require full-pose commands
and sometimes require position-only commands, depending on the script. Each of the scripts
explicitly set the command mode using
tools.commander.set_target_full_pose() # either this
tools.commander.set_target_position_only() # or this
when the behaviors are built. If you examine the implementations of
set_commander_to_full_pose.py
and set_commander_to_position_only.py
, you’ll find that those
methods are used under the hood.