User Manual¶
Hot Keys¶
Key |
Action |
---|---|
P |
Play |
R |
Rewind |
L |
Loop |
[ |
Set Range Start |
] |
Set Range End |
Audio Player and Recorder¶
Allows you to load custom audio and serves as the primary playback mechanism for Audio2Face.

Option |
Effect |
---|---|
Audio source Directory |
Browse to a folder destination to load all of the audio files within that folder into the “track” menu.
|
Track |
Allows you to select which audio track you would like to playback from your specified library location. Only .wav files are supported currently.
|
Range |
The first box allows you to clip (trim) an audio file to set a desired start time. The second box allows you to clip the audio to specify the end point.
|
Timeline/AudioWaveForm |
A visually descriptive timeline of the audio track - Use mouse to scrub back and forth.
|
Play Controls |
Play - Return to Head - Loop playback and Record.
|
Recording |
Pressing the record button (red Dot) in the play controls panel will reveal the recording tools and live mode for audio2face.
|
Mute |
stops the audio stream to A2F.
|
Rec |
Click to record a new Wav audio file
|
Live |
Click to enable live mode - A2F driven by real time voice data input.
|
New |
Creates a new .wav file to record - enter desired name in the panel to the right. Auto increments names when new tracks created with the same name.
|
Save |
Allows you to save your audio recording as a .wav file.
|
Attached Meshes¶
Allows the selection of the desired source training asset.

Nvidia’s Audio2Face asset loads by default.
This feature will allow you to assign custom meshes for custom A2F neural networks (NN). (coming soon)
Network¶

Option |
Effect |
---|---|
Network Name |
Select which Neural Network you want to use.
|
Processing Time |
Displays the latency of the selected Network.
|
Pre Processing¶
Allows the adjustment of key variables that influence the animation result.

Option |
Effect |
---|---|
Prediction Delay |
Allows the user to tweak the alignment of the audio input and animation output.
|
Input Strength |
Adjusts the audio volume - allowing the user to influence the range of motion of the animation resolve.
Can be muted for a subtle performance or amplified for larger/louder performances.
|
Post Processing¶
Allows the user to independently control smoothness and strength of the motions on upper and lower face.

Option |
Effect |
---|---|
Upper and Lower Face Smoothing |
Smooths the motions on the upper and lower regions of the face.
|
Upper and Lower Face Strength |
Allows the user to determine the amplification of the motion on the upper and lower regions of the face.
|
Face Mask Level |
Allows the user to determine the boundary of the upper and lower region of the face. (in the axis of the height)
|
Face Mask Softness |
Allows the user to determine how smoothly the upper and lower face regions blend on the boundary.
|
Emotion¶
Allows the user to select a specific face shape from animation training sources to impose on the default face - allowing the user more control to inject specific emotional expressions onto the performance.

Option |
Effect |
---|---|
Source Shot |
Selects the animation clip.
|
Source Frame |
Selects the specific frame from the animation source to use as shape of influence.
|
Multiple instances of Audio2Face¶
You can create multiple instances of Audio2face to run multiple characters in the same stage / scene.

Option |
Effect |
---|---|
|
Adds a connected set of A2F modules.
|
|
Adds another Default head asset to the scene.
|
|
Creates a new Audio Player.
|
|
Creates a new instance of the Audio2Face Network.
|
OmniGraph Editor¶
Each Audio2Face instance, audio player and target geometries relationship can be changed in Omnigraph to allow flexible arrangements of which Audio player is connected to which character / instance of audio2face.

Streaming Audio Player¶
Streaming Audio Player allows streaming audio data from external sources and applications via gRPC protocol. The player supports audio buffering. Playback starts automatically.

To test Streaming Audio Player you can open corresponding demo scene in the image below.
Menu -> Audio2Face -> Open Demo Scene -> Streaming Audio Scene.

Or create a player manually
In the Audio2Face panel click “+ Audio Player” -> “+ Streaming Audio Player”.
Or, alternatively, Menu -> Audio2Face -> “+ Audio Player” -> “+ Streaming Audio Player.

Once the player is created, you need to connect it to the Audio2Face Core instance in Omni Graph Editor (connect corresponding “time” attributes).

Use the Prim Path of the created player as one of the parameters for the gRPC request, when passing audio data to the player from an external source.

To send the audio data to the player you need to implement your own logic (client), which pushes gRPC requests to the Streaming Audio Player server.
Example of such client implemented in Python is provided in “test_client_py”, which is located in A2F directory.
AUDIO2FACE_ROOT_PATH/exts/omni.audio2face.player/omni/audio2face/player/scripts/streaming_server/test_client.py
The script provides detailed description of the protocol and parameters (see comments) and also serves as a simple demo client, which takes input WAV audio file and emulates an audio stream by splitting it into chunks and passing it to the Audio Player.
To test the script: - Launch Audio2Face App and open the Streaming Audio Scene (or create a player manually) - Run the test_client.py script: - python test_client.py <PATH_TO_WAV> <INSTANCE_NAME> - PATH_TO_WAV: provide a path to any WAV audio file with speech - INSTANCE_NAME: Streaming Player Prim Path (see above)
Required pip packages to be installed. - grpcio - numpy - soundfile - google - protobuf
Character Transfer - Retargeting Pipeline¶
The Mesh Retargeting tool is available on the Character Transfer Tool tab.

The status message at the bottom of the widget informs the user if there are any steps missing to execute the retargeting successfully.
You can access sample assets used in the online tutorials for the character transfer process in the following locations.
Omniverse mount:
\Nvidia\Assets\Audio2Face\Samples\
\<Audio2Face Install Directory>\assets\
Note
Hidden directories are currently not visible in content browser / the app’s file browser - will be fixed in upcoming version)
Meshes¶
In this panel, users can select the meshes to use in the character transfer process.

Option |
Effect |
---|---|
Driver A2F Mesh |
Selects the character mesh that will be driven by A2F. It should be in neutral closed mouth expression.
This mesh is not used for picking correspondence points.
(On mark retarget example, this would be /World/charTransfer/mark)
|
Target Mesh |
Selects which character head to retarget onto.
|
After the target mesh is assigned the “openMouthA2Fmouth” field will appear.
Option |
Effect |
---|---|
OpenMouth A2F Mesh |
Selects the character mesh that share the same topology as the Driver A2F Mesh, but in Open mouth expression.
(On mark retarget example, this would be /World/charTransfer/mark_openMouth)
|
Mesh Fitting - Correspondence¶
This is the authoring panel for the correspondence points. This panel remains minimized until the meshes are correctly assigned in the Meshes panel.

Option |
Effect |
---|---|
Correspondence visibility |
Visibility switch for the points and lines correspondence viewport draw.
|
Preset |
Load and save correspondence setups for custom characters and preloaded presets.
|
Display Size |
Controls the draw size of the correspondence points.
|
Correspondence list |
This widget shows the authored correspondences. Users can select the correspondence pair from the list by clicking it.
Shift key or Ctrl can be used in combination with the mouse click to do multiple selections.
|
Add Mode |
Enter Add Mode. In this mode, the user can pick the correspondence points between the meshes.
|
Edit Mode |
Enter Edit Mode. In this mode, the user can reposition the existing correspondence points by clicking and dragging
the point to a new position on the mesh.
|
Delete Mode |
This will delete selected correspondence points on the list.
|
Fitting Iterations |
Shape Fitting number of iterations. Can be more accurate with a higher number but also takes longer to execute.
3 is a good starting point and is used on all our example characters.
|
Fitting Stiffness |
Shape Fitting stiffness to try keep its original contour.
100 is a good starting point and is used on all our example characters.
|
Begin Mesh Fitting |
Execute the retarget operations when clicked.
|
Post Wrap¶
Post wrap panel shows the available fitted mesh for the post wrap process. The Smooth Level and and Begin Post Wrap button will be hidden if not Fitted Mesh is selected from the drop down.

Option |
Effect |
---|---|
Fitted Mesh |
Fitted mesh is assigned in this field. Mesh created from Mesh Fitting process above will
be automatically selected when the Mesh Fitting process finishes.
|
Smooth Level |
Smoothing level on the target mesh. Low number preserves the original details of the A2F mesh,
high number smooths out the details and possible artifacts - Recommended to start low and increase as needed.
|
Begin Post Wrap |
Start Post Wrap process as part of retarget. This process applies multiple deformers that are
chained to make the live connection between a2f mesh and the target mesh.
|
Tools¶

Option |
Effect |
---|---|
|
Adds A2F neutral “mark” head and “mark_openMouth” meshes to the stage. Both necessary for the character transfer pipeline.
|
Delete setup |
Removes all Character Transfer related nodes and connections as well as reset the meshes to a2f neutral state.
|
Delete Correspondence |
Deletes all user created correspondence points assigned to the meshes.
|
Wrap UI |
Opens Wrap UI which offers additional options for the wrapping process.
|
Wrap UI¶
Wrap UI provides some options for users to use one driver mesh to drive one or more secondary meshes. For example, the face mesh is used as a driver for the eyebrows and moustache meshes. This UI execution uses the scene selection information to know which meshes to execute on. The first selected mesh would be the driver to drive one or more of the following selected meshes.

Option |
Effect |
---|---|
Use Max Distance |
Toggle for considering Max Distance field below to set the weights of the deformer.
When enabled, any driven points further than the Max Distance to the driver mesh
will have 0 weight, hence not affected by the Wrap deformer.
|
Max Distance |
The cutoff distance of each driven point to the driver mesh surface. This field is
disabled until the Use Max Distance checkbox above is enabled.
|
Face Tangent Space |
Use face tangent space to store and apply the offset vector of each point to the
surface. When disabled, the offset vector would be applied in the object space
of the driven mesh.
|
When the tool detects the intended driven mesh already has an incoming connection, a popup would appear for each mesh to let the user decide the next action.
Option |
Effect |
---|---|
Skip |
Leave the specified mesh be. No new Wrap deformer would be created for this particular mesh.
|
Replace |
Delete the upstream node of the mesh, and replace it with the new created Wrap deformer.
|
Add |
Add new Wrap deformer and connect it to the mesh. The previous upstream node will be
disconnected from the driven mesh but not deleted.
|
BlendShape Generation¶
The Blendshape Generation tool allows the user to generate a set of blendshapes from a custom neutral mesh. Run your mesh through the character Transfer process, select your mesh then click “Blendshape Transfer”. A full set of shapes will be generated and available for export as USD for use in any DCC application.

Option |
Effect |
---|---|
BlendShape Transfer |
Runs the process to generate a set of Blendshapes for a target Head mesh.
|
Export as USD |
Exports the blendshapes as USD for use in other applications.
|
Common Settings¶

Option |
Effect |
---|---|
Export Directory |
Select a destination for the point cache export.
|
File Name |
Specify the filename of the exported cache.
|
Playback FPS |
Select the Frames per second you would like the cache to be exported at.
|
Geometry cache¶

Option |
Effect |
---|---|
Export Filter |
Select the mesh filter to select the meshes within the stage from which you want to export the animation cache for. (only exports the animation data - does not export the source mesh.)
|
Export as |
You have the choice of a USD point cache export or a Maya point cache export.
|
Additional Export Notes¶
When exporting a custom mesh used in the character transfer process. Be sure to select the mesh in the stage that is labelled with “_result”.
When exporting a Maya cache, you can import the source mark head to Maya to assign the cache too. A mark asset (male_head_model_hi_published.mb) can be found in your install directory /audio2face/assets/
Blendshape Conversion¶
Allows the user to convert the output a2f animation into a blendshape-driven animation

Option |
Effect |
---|---|
Input anim mesh |
The mesh you want to convert into a blendshape animation. It is usually the mesh from the A2F pipeline.
|
Blendshape mesh |
The neutral mesh with blendshape prims and this should be based on the usdSkel schema.
|
Set up blendshape solve |
Setting up an interactive blendshape solve pipeline in the stage.
|

Export as json |
Export the resulting blendshape weight values as .json file.
|
Export as USD SkelAnimation |
Export the resulting blendshape weight values as .usd in the SkelAnimation prim.
|
Additional notes |
When exporting animation, you will use the information in “common settings” for the file path and the playback fps.
|
MetaHuman |
Now supports Unreal Engine’s MetaHuman via the Omniverse Unreal Engine Connector version 103.1. The exporter creates custom “animation curves” that are imported as an animation sequence to align with the Metahuman skeleton.
|
Two arrays |
|
custom:mh_curveNames |
This token array contains the names of the control curves, like CTRL_expressions_eyeLookUpR.
|
custom:mh_curveValues |
this float array contains the time sampled curve value data.
|
Pose & pose option table¶
Option |
Effect |
---|---|
Pose |
Name of the blendshape poses.
|
Use |
whether we want to include or exclude the pose in the blendshape solve.
|
Cancel pair |
tagging 2 poses that cancel each other. They cannot be used together at a frame, only one of them can get a non-zero weight.
|
Symmetry |
tagging 2 poses that are symmetric. They try to have the same weights during the blendshape conversion.
|
Select All, Selection options |
(Right mouse click menu): Handy pose selection options.
|

Toggle use |
Toggle the selected “use” checkboxes in the table.
|
Tag/untag cancel pair |
Tag/untag the selected 2 poses as a cancel pair.
|
Tag/untag symmetry pair |
Tag/untag the selected 2 poses as a symmetry pair.
|
Weight regularization |
Stronger value prevents the solve to compute strong weight values. Weight regularization.
|
Temporal smoothing |
Stronger value prevents pops between consecutive frames. Temporal weight regularization.
|
Weight Sparsity |
Stronger value generates sparse blendshape weights, which is preferable for animator’s edit. L1 weight regularization.
|
Symmetric Pose |
Stronger value makes the symmetry poses to have more similar weight values.
|
Load preset |
Load preset blendshape options from a .json file.
|
Save preset |
Save blendshape options as preset in a .json file.
|
Batch Audio Processing¶
Allows the user to export blend weights as .json or .usd, for all of the audio files within the specified Audio Source Directory.

Option |
Effect |
---|---|
Blendshape Weights Json |
Select to export as .json
|
Blendshape Weights USD option |
Select to export as USD
|