Project Name

·         HIK: A hybrid inverse-kinematics solution to generate avatars in real time for virtual reality applications.

Website

·         https://www.linkedin.com/in/mathew-wang-6497b854/ , the author's linkedIn page.

·         https://homepage.divms.uiowa.edu/~kearney/ , the homepage of the research advisor, project supervisor, Dr. Joseph K. Kearney.

Project Demo Videos:

·         A demo of HIK working on different models, in the sequence of their presence they are: 163 joints model from Makehuman.org, 63 joints model from mixamo.com, a model of Metahuman, 67 joints model from mixamo.com .

·         A demo of the full body movement comparison between the virtual person and the real person, HIK (the hybrid inverse kinematics method), works on a general (fully rigged) humanoid model, regardless of skeletal structure or bone convention, allows small adjustment for body height and arm stretch length to adapt for different human subjects.

·         A demo of the full body movement comparison between the virtual person and the real person. The movement is driven by HIK.

·         A demo after implementing "parallel restart": numerical IK algorithms get stuck at local minima more often with the defined joint limits, robotic IK has numerous solutions to restart IK search, inspired from robotic IK, HIK uses the posture graph search to find local minima to initiate numerical IK computation in PARALLEL, that reduces chances of a failed numerical computation, in practice, this solution has fail rate lower than 1%. Numerical IK computation restart could cause jittering and jerky movement, choosing from several possible IK solutions for the smoothest posture transition, could reduce the side effect.

·         A demo of the GSIK (generic sequential IK framework) and SDLS IK chain solver with a posture graph, without joint limit specification, generates natural postures and smooth transitions.

·         A demo of the GSIK (generic sequential IK framework) and SDLS IK chain solver without a posture graph and joint limit specification can generate an unnatural posture.

·         https://youtu.be/lHIGJTb1be8, BVH file play back in Unreal Engine 4 through animation retargeting.

·         https://youtu.be/1HVC7dCYBzU, the comparison between HTR and BVH.

·         https://youtu.be/gXLk47gTAGY, a prototype of IK usage in VR environment with Blender SDLS IK solver.

·         https://youtu.be/C-mCLFqCMOA, a prototype of IK with DLS (damped least square) method with MATLAB.

·         https://youtu.be/bSer7F_VfOE, IK Error visualization through per-vertex shading in Unity3D platform.

Link to project build

·         https://github.com/wwxhero/lib_hIK.git , a library exposing the forward kinematics and inverse kinematics functions, the library interface is in C language, the implementation is with C++ language.

·         https://github.com/wwxhero/hIK_unreal.git, an animation node plugin that utilizes the lib_hIK for forward kinematics or FK, and inverse kinematics or IK, for FK, it supports playing back BVH file, a motion capture data format, within the Unreal animation system, for IK, it takes configurable number of end effectors as input, and generate the local space bone configurations. It interfaces with lib_hIK through a simple set of C functions.

·         https://github.com/wwxhero/hIK.git, a set of unreal engine game projects for testing and using the IK (inverse kinematics) in Unreal and VR environment.

·         https://github.com/wwxhero/Posture_Graph, a set of tools that work on BVH files and HTR files (not standard HTR files, the header part of file is in BVH, the animation stack stores HTR animation, since there is no free tool that supports HTR, thus for convenience this semi-HTR is chosen to be used.), to build the posture graph, more information about posture graph in the project description section.

·         The classes design overview in UML(hit for a large view):

Chart, line chart Description automatically generated

Project Type

·         Education

Project Description

This is a Ph.D. dissertation project, the goal of this project is to create a full-body inverse kinematics (IK) solution that generates postures in real time in a VR environment powered with the Unreal Engine 4. The input includes 6 tracking locations: head, pelvis, wrists, and ankles. The key points of the goal include:

1.       The IK solution generates postures in real time and the quality meets standards for both computation speed and fidelity. For the computational speed standard, it should not drop frames per-second (FPS) lower than 50 for a normal scene setting; for the fidelity standard, the posture it generates should look natural and approximate the real posture, and the posture transitions or animation should be smooth.

2.       The IK solution generates postures for any humanoid model rigged with a general skeletal structure, such as models from http://www.makehumancommunity.org/content/downloads.html or from the Metahuman creator for Unreal Engine 4.  

To achieve the first goal, the numerical IK methods are favored. Damped least square or DLS and selective damped least square or SDLS are used for animation making that generates smooth transitions. The smooth and continuous input of the VR tracking data is also a positive factor for the DLS and SDLS solvers to work fast to meet the speed requirement standard. To meet the fidelity standard, we use a posture graph. A posture graph  is constructed from a set of postures where, , collected from BVH files converted from the CMU web site, http://mocap.cs.cmu.edu/tools.php and https://sites.google.com/a/cgspeed.com/cgspeed/motion-capture/cmu-bvh-conversion. The files store millions of postures, and  are the (natural) posture transitions. The body movement is best described with a graph, where each posture is a snapshot of the body movement and transitions are for the posture change. Searching along the natural transitions for a posture that best satisfies the end effectors orientation and position would result a smooth and natural animation. This argument will be shown true through this project; a conventional method with the joint limit will work as a supplement to the posture graph to guarantee posture naturalness. L-BFGS, a gradient based optimization method, will be added for a better performance with joint limits, if necessary, at the end. Thus, it requires to have different numerical IK methods working under a generic sequential IK framework or GSIK that breaks the humanoid skeletal into kinematic groups, where each group includes at least one kinematic chain and each numerical IK algorithm works on a chain. For dealing with the performance uncertainty, we provide the configurations for defining the IK chains and individual joints. The parameters to tune the IK algorithm performance is configurable through an XML file. An example of a configuration XML file is shown below,

Using XML could avoid the impact from Unreal Engine upgrades. The configuration works as a supplement to the source code, but without the need to compile and build. We optimized the forward kinematics (FK) that is fundamental for IK: rotation is represented with quaternions giving the advantage of no scaling in the skeletal structure of BVH and HTR. FK computations can run as fast as a million times per second. FK results are cached until the posture changes. FK is frequently used for IK computation, thus IK is optimized as well. At the integration layer for Unreal Engine 4, we optimized the animation node output for the Unreal Engine 4 animation system. Since HIK is defined as a full body IK solution, the animation node of the Unreal Engine 4 powered by HIK outputs a local space “Transform”, unlike the Unreal Engine 4 built-in IK that generates the component space “Transform” that fits for a small number of bones (1-2 bones) adjustment. HIK generates postures of an avatar with many more bones. It is not uncommon to see an avatar with more than 100 bones. Working in this way directs the output posture into the Unreal Engine animation system without any unnecessary intermediate animation processing.

Chart, line chart Description automatically generated

Figure 1. The animation blueprint with built-in CCDIK animation node with an intermediate processing.

Figure 2. The animation blueprint with an HIK animation node direct output in local space.

 

To achieve the second goal, we use animation retargeting. A posture graph is built on the skeletal structure of 31 bones using the HTR convention: the Y axis is aligned with the bone axis. To support an avatar with any number of bones possibly without this convention, we retarget the IK result from the 31-HTR-bone structure to the unspecified humanoid model with animation retargeting. The animation retargeting is defined with pairs of bones in XML file. This mechanism enables a wide support of humanoid models, with only a little of configuration work. A snippet of code for animation retargeting definition in an XML file is shown below.

The IK solution shares some properties in common with the formerly available commercial full-body IK product, Ikinema. Both can specify the number of iterations and the weight of position and orientation tasks. Since only the numerical IKs use the weight for position and orientation tasks, we assume that Ikinema is also based on numerical methods.  Both methods generate a local space ‘Transform’ for the Unreal Engine animation system. But HIK differs from Ikinema in the following respects:

1.       the HIK source code is open under the GNU General Public License.

2.       HIK supports a general unreal engine skeletal mesh through animation retargeting. Ikinema uses Unreal Engine 4 UI operations to configure for each model.

3.       HIK uses a posture graph to boost numerical computation.

HIK ports the DLS and SLDS numerical solvers from Blender code with the following additions and extensions:

1.       Blender IK computation uses double precision data types. The goal of IK is to generate smooth transitions but not for real-time computation. HIK is more flexible on the data type, which could either be defined as float or double for a real number data type.

2.       HIK numerical computation is done with a kinematic-group coordinate space as the world space. This results in the translation part of transformation being compacted around zero or the origin which is important for the real number to be defined as a single precision floating point number with a limited precision. Group-space coordinates best exploit the floating-point number precision limit.

The progress of the project is steady, here is the list of work steps to achieve our goal:

  1. Prototype IK usage in VR environment with Blender.
  2. BVH data parsing and integration into Unreal Engine 4 animation system.
  3. BVH to HTR conversion.
  4. Articulated Body tree structure is established and being used for FBX, BVH, HTR data.
  5. IK testing environment is set on Unreal Engine 4.
  6. Articulated body partition for joint groups, with each group including at least 1 joint chain.
  7. A numerical IK solver that works on a joint chain.
  8. GSIK + numerical chain-solver (version 1.0).
  9. Construct posture-graph from HTR data for each joint group.
  10. Integrate posture-graph search into HIK solution and enable parallel computation (version 2.0).
  11. (Optional) Mining BVH data to study the joint stiffness and joint rotational limit parameters.
  12. Evaluate IK solution with different parameter settings in a mock VR environment (version 2.1).
  13. Evaluate IK usage in VR environment and make a final demo.
  14. Make available for more model choices, this solution is designed to support a generic articulated model (FBX) that can posed in 'T' posture.
  15. To fix the minor issue: shoes penetrating deep into ground.

At present, the work has been completed through item 14.

Posture Graph fundamentals, the epsilon postures (left) and the corresponding posture graph visualizations:

Chart, line chart Description automatically generated

Figure 3. The errors between postures highlighted in red are less than epsilon (2 degrees) .

Figure 4. Posture graph generated from a single dissected HTR file with epsilon=2, |V|=848,|E|=933 Average degree is 2.2 Network diameter is 106 Average path length is 34.248, rendered with 'Gephi'.

Joint Limit Specifications and Visualization

Joint limit specification is through XML configuration file, the numerical algorithm clamps the joint rotation within the limited range, to avoid the extream cases that violate joint bio-mechanical possibilities, the following snipped of XML code is an example of joint limit specification:

Joint rotation is first decomposed as swing-twist components: twist component is named as "R_tau", swing component is named as (R_theta, R_phi) respectively, they are all in degress, with different ways for clamping, intepreting (R_theta, R_phi) can be different, C_Sphercial clamping interprets (R_theta, R_phi) as sopherical coordinate (ro, theta, phi) with ro = 1, direct clamping interprets (R_theta, R_phi) as the two components for quaternions (_, sin(alpha/2)*u_x, _, sin(alpha/2)*u_z) where u = (u_x, 0, u_z) is the unit rotational vector, and alpha is the rotational angle. for the joint (knee) with one DOF rotation, specifying rotation with direct clamp is more intuitive, for the joint with three DOFs rotation, sopherical coordinate will perform better along the clampping edges, it can be mathematically proven. Twist rotation is intuitive to understand, thus only the swing rotation is visualized. Comparing the two figures below, we can see the spherical map has a steighter edges, thus more suitable to clamp with in-equations.

Chart, line chart Description automatically generated

Figure 7. the swing rotations LeftArm Joint in direct map visualization.

Figure 8. the swing rotations LeftArm Joint in spherical map visualization.

Major events with the time line

·         Pass the thesis defence Exam (Hooray) (April, 2022)

·         Integrting posture graph runtime search with the nuermical IK solutions, with support for joint limit, joint dexterity, parallel IK search with posture graph, and nuermical computation restart. (Dec, 2021 - Jan, 2022)

·         Build posture graph solutions (runtime+offline). (Oct, 2021 - Dec, 2021)

·         Implementing GSIK (general sequential IK) framework, porting Jacobian nuermical solver from Blender, and re-structuring, integrating into GSIK framework. (Sep, 2021)

·         Pass the proposal defence Exam (Hooray) (August, 2021)

·         Re-orient bone axis with HTR convension for the BVH data sources. (July, 2021)

·         Setup FK support with Eigen library by implementing BVH animation and retargeting to Unreal sleltel mesh on Unreal Engine 4. (June, 2021)

·         Prototype IK usage in VR environment with Blender. (May, 2021)

Target platforms

·         Windows + Unreal Engine 4 + HTC VIVE.