AI Motion Capture

Estimated reading: 4 minutes

Introducing the “AI Motion Capture” project type, where getting the animation data from any video has never been easier. With just a few clicks, you can effortlessly transform your video into Motion Capture data. Upload your footage, scan any frame for actors, assign CG characters to the actors you wish to get the animation data for, and simply click “Start Processing”. Sit back, relax, and let our advanced AI handle all the heavy lifting for you.

The “AI Motion Capture” process consists of 3 main steps:

AI Motion Capture Project Guidelines

AI MoCap delivers character animations in the camera space. This means that the character’s motion will be accurate in reference to the camera. If the camera itself is moving through space, its motion will have to be tracked separately to get the character’s animation in world space.

MetaHuman AI Motion Capture Integration

Processing

Prerequisites:

AI Mocap Project Type Only

Overview:

The new Wonder Character feature is designed exclusively for the AI MoCap project type. It enables users to generate out-of-the-box animation data that is fully compatible with MetaHuman Characters, which can be used in Unreal Engine Projects.

Bringing the Wonder Character into Your Project:

Like any other Character in Flow Studio, you can select the MetaHuman poster for the character and drag and drop it onto the Actor you’d like to generate motion capture data for.

Custom Options Configuration:

Within the Custom Options modal (left-hand side), you’ll now see a new configuration option: MetaHuman Body Type.
From here, you can select:
- Gender
- Height
- Weight
- For the best results, we recommend selecting the exact MetaHuman body type that matches the one used in your Unreal Engine project.
  - MetaHuman skeletal meshes, imported from Quixel Bridge, can be located in the MetaHumans/Common folder in the format above

Note: If you’re unsure of the exact body type, or you pick the wrong body type, height or weight during setup, do not panic! The animation data can still be salvaged using Unreal Engine’s built-in retargeting tool.

Import

Prerequisites for Importing into Unreal Engine:

Go to EDIT –> PLUGINS –> USD Importer

Overview:

Once processing is complete, you can download the output as a USD asset from the AI MoCap.

Importing the USD Asset:

Drag and drop the generated Actor.usda file into Unreal Engine.
The USD Importer modal will appear automatically.
In the USD Importer modal:
1. 1. Left Side (Prims Selection)
    1. You can select which prims to import, but for a full import, you should leave all items checked.
1. Right Side (Stage Options)
  1. Check “Override Stage Options”
  2. Then, set the following parameters in the “Stage Options” dropdown:
    - Meters Per Unit: 0.01
    - Up Axis: Z Axis

Setup

Prerequisites for Setup in Unreal Engine:

Go to EDIT –> PLUGINS –> USD Importer

Overview:

After importing the USD asset, follow these steps to finalize animation setup for your MetaHuman character:

Open the Animation Asset
1. Locate and open the imported Animation Instance Asset, typically named AS_Anim
Set the Preview Mesh
1. On the right side of the Animation Asset window under Preview Scene Settings, update the Preview Mesh to your desired MetaHuman skeletal mesh.
  1. You can use any MetaHuman skeleton imported via Quixel Bridge (This ensures the animation data aligns correctly with MetaHuman skeletons in Unreal Engine).
  2. Prerequisite:
    - In the Preview Mesh search modal, click the settings icon and make sure “Allow Incompatible Skeletons” is enabled
2. Once you change the mesh, the Animation Asset window will automatically reload and display a new preview with the selected mesh.
Set the Retarget Source
1. On the left side of the Animation Asset window under Asset Details, set the Retarget Source Asset to the MetaHuman body type that matches the one selected in your AI MoCap project
  1. For example: m_med_nrw_body
Load into a Level Sequence
1. After configuration, you can load the animation directly into your desired Level Sequence in Unreal Engine.