Accelerate Large-Scale Inverse Kinematics with the Intel® Distribution of OpenVINO™ Toolkit
2 likes5,511 views
The document discusses the application of deep learning-based inverse kinematics (IK) for character animation in games, utilizing the Intel OpenVINO toolkit for optimization. It emphasizes the advantages of this approach over traditional methods, including improved performance and flexibility. Key topics include the architectural design of the IK solver, the process for gathering training data, and performance insights on CPU versus GPU inference.
![@IntelSoftware @IntelGraphics 11
Architecture [1/2]
Trajectory
Network
60 Pose
Networks
R
R
5 Trajectories
Deep Learning based IK Solver](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-11-320.jpg)
![@IntelSoftware @IntelGraphics 12
Architecture [2/2]
Trajectory
Network
60 Pose
Networks
Curve
fitting
R
R
R
R
5 Trajectories
Smoothened
5 Trajectories
Deep Learning based IK Solver](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-12-320.jpg)
![@IntelSoftware @IntelGraphics
① Small-sized tasks
16
CPU vs GPU [1/3] : Requirements
Inference for IK
• Small-sized neural networks
• Tasks are performed for each
game loop
• Small batch size
• Require quick response
( low–latency )
• GPU is busy rendering.
② Inference on Game-Client
Client](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-16-320.jpg)
![@IntelSoftware @IntelGraphics 17
CPU vs GPU [2/3] : ① Small-sized tasks
Small sized neural networks and
small batch size
Large sized neural networks and
large batch size](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-17-320.jpg)
![@IntelSoftware @IntelGraphics 18
CPU vs GPU [3/3] : ② Inference overhead on Game-Client
Without Inference CPU usage GPU usage Game frame rate Inference latency
No Game - - - -
Overwatch 22 % 97 % 146 fps -
PUBG 38 % 97 % 141 fps -
Assassin Creed : Odyssey 53 % 96 % 69 fps -
Inference on CPU using
Openvino CPU usage GPU usage Game frame rate Inference latency
No Game - - - 50.63ms
Overwatch 68 % ( 309.1 % ) 98 % ( 101.0 % ) 145 fps ( 99.3 % ) 57.97 ms ( 114 % )
PUBG 75 % ( 197.4 % ) 96 % ( 99.0 % ) 137 fps ( 97.2 % ) 59.92 ms ( 118 % )
Assassin Creed : Odyssey 84 % ( 158.5 % ) 96 % ( 100.0 % ) 69 fps ( 100.0 % ) 82.87 ms ( 164 % )
CPU : Intel i7-6700K GPU : Nvidia GTX 1080 ( ) : Rate of change
Inference on GPU using
Openvino
CPU usage GPU usage Game frame rate Inference latency
No Game - - - 43.87ms
Overwatch 38 % ( 172.7 % ) 97 % ( 100.0 % ) 119 fps ( 81.5 % ) 77.51 ms ( 177 % )
PUBG 66 % ( 173.7 % ) 97 % ( 100.0 % ) 113 fps ( 80.1 % ) 76.01 ms ( 173 % )
Assassin Creed : Odyssey 66 % ( 124.5 % ) 96 % ( 100.0 % ) 58 fps ( 84.1 % ) 88.93 ms ( 203 % )](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-18-320.jpg)
![@IntelSoftware @IntelGraphics 21
Optimization : Cubic Hermite Spline [1/2]
! = 0
! =1
$%
$&
'%
'&
$ ! = 2!) − 3!, + 1 $% + !) − 2!, + ! '% + −2!) + 3!, $& + (!) − !,)'&](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-21-320.jpg)
![@IntelSoftware @IntelGraphics 22
Optimization : Cubic Hermite Spline [2/2]
! " = 2"% − 3"( + 1 !+ + "% − 2"( + " ,+ + −2"% + 3"( !- + ("% − "(),-
! " = "% "( " 1
2 −2 1 1
−3 3 −2 −1
0 0 1 0
1 0 0 0
!+
!-
,+
,-
Matrix calculations can be accelerated by Intel Openvino .
in matrix form](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-22-320.jpg)
![@IntelSoftware @IntelGraphics
IK Solver
24
Optimization for multiple characters [1/2]
Character
Character
Character
Character
Character
Motion timer
updates frame number.
Trajectory
Network
Pose
Network
Next frame motion data
Batch Manager
Request next frame
motion data
Next motion frame](https://image.slidesharecdn.com/08-accelerate-large-scale-inverse-kinematics-with-the-intel-distribution-of-openvino-toolkit-190405185819/85/Accelerate-Large-Scale-Inverse-Kinematics-with-the-Intel-Distribution-of-OpenVINO-Toolkit-24-320.jpg)
Accelerate Large-Scale Inverse Kinematics with the Intel® Distribution of OpenVINO™ Toolkit
- 1. Yoon Dongwon (Game AI Team, Game AI Lab, AI Center, NCSOFT) Jang Hanyoung (Motion AI Team, Game AI Lab, AI Center, NCSOFT)
- 2. @IntelSoftware @IntelGraphics 2 Agenda • Introduction • Deep learning based Inverse Kinematics • Optimization using Intel® OpenVINO™ toolkit • Result • Conclusion
- 3. @IntelSoftware @IntelGraphics 3 The Team Game AI Lab Motion AI Team Game AI Team Reinforcement Learning Team
- 5. @IntelSoftware @IntelGraphics 5 Inverse Kinematics (IK) • Compute joint angle that makes end-effector move to the target. Target Joint angle 3 Joint angle 1 Joint angle 2 End-effector ? ? ? Catch the target
- 6. @IntelSoftware @IntelGraphics 6 Inverse Kinematics in Games • Generate animations that interact with the surrounding environment ex) Solving the foot skating problem ex) Generate animation to catch some object
- 7. @IntelSoftware @IntelGraphics 7 Goal • Large number of characters climb rugged cliff with IK
- 8. @IntelSoftware @IntelGraphics 8 Challenges Traditional methods are not suitable for the objective Numerical approach Analytical approach Quality Good Bad Performance Slow Fast Numerical approach Analytical approach Deep Learning Quality Good Bad Good Performance Slow Fast Fast Solution
- 10. @IntelSoftware @IntelGraphics 10 Approach “Input” • Current Hands and Feet Positions • Target Hands and Feet Positions “Output” • Climbing motion • Number of Frames : 60 • Length : 2 seconds Deep Learning based IK Solver R R
- 11. @IntelSoftware @IntelGraphics 11 Architecture [1/2] Trajectory Network 60 Pose Networks R R 5 Trajectories Deep Learning based IK Solver
- 12. @IntelSoftware @IntelGraphics 12 Architecture [2/2] Trajectory Network 60 Pose Networks Curve fitting R R R R 5 Trajectories Smoothened 5 Trajectories Deep Learning based IK Solver
- 13. @IntelSoftware @IntelGraphics 13 Gathering Training Data 1. Produce manually created “reference motion”. 2. Then randomize “start and end targets” based on “reference motion”. 3. Finally for each “start and end targets”, generate motion data using “numerical approach”. Random targets Reference motion Generated motion using numerical approach
- 15. @IntelSoftware @IntelGraphics 15 Overview Training server Infers IK Animation Deploy Trained Network “How to optimize client-side inference?” • The process of Deep-learning based IK Client
- 16. @IntelSoftware @IntelGraphics ① Small-sized tasks 16 CPU vs GPU [1/3] : Requirements Inference for IK • Small-sized neural networks • Tasks are performed for each game loop • Small batch size • Require quick response ( low–latency ) • GPU is busy rendering. ② Inference on Game-Client Client
- 17. @IntelSoftware @IntelGraphics 17 CPU vs GPU [2/3] : ① Small-sized tasks Small sized neural networks and small batch size Large sized neural networks and large batch size
- 18. @IntelSoftware @IntelGraphics 18 CPU vs GPU [3/3] : ② Inference overhead on Game-Client Without Inference CPU usage GPU usage Game frame rate Inference latency No Game - - - - Overwatch 22 % 97 % 146 fps - PUBG 38 % 97 % 141 fps - Assassin Creed : Odyssey 53 % 96 % 69 fps - Inference on CPU using Openvino CPU usage GPU usage Game frame rate Inference latency No Game - - - 50.63ms Overwatch 68 % ( 309.1 % ) 98 % ( 101.0 % ) 145 fps ( 99.3 % ) 57.97 ms ( 114 % ) PUBG 75 % ( 197.4 % ) 96 % ( 99.0 % ) 137 fps ( 97.2 % ) 59.92 ms ( 118 % ) Assassin Creed : Odyssey 84 % ( 158.5 % ) 96 % ( 100.0 % ) 69 fps ( 100.0 % ) 82.87 ms ( 164 % ) CPU : Intel i7-6700K GPU : Nvidia GTX 1080 ( ) : Rate of change Inference on GPU using Openvino CPU usage GPU usage Game frame rate Inference latency No Game - - - 43.87ms Overwatch 38 % ( 172.7 % ) 97 % ( 100.0 % ) 119 fps ( 81.5 % ) 77.51 ms ( 177 % ) PUBG 66 % ( 173.7 % ) 97 % ( 100.0 % ) 113 fps ( 80.1 % ) 76.01 ms ( 173 % ) Assassin Creed : Odyssey 66 % ( 124.5 % ) 96 % ( 100.0 % ) 58 fps ( 84.1 % ) 88.93 ms ( 203 % )
- 19. @IntelSoftware @IntelGraphics 19 Choosing Solution for Deep Learning Inference 423.48 ms 34.75 ms 20.34 ms 2.21 ms 0 50 100 150 200 250 300 350 400 450 Naive cpp Numpy Tensorflow DL Inference Engine Average Latency on CPU (Lower numbers indicate better performance) 12X faster 20X faster 191X faster
- 20. @IntelSoftware @IntelGraphics Game Client IK Solver 20 Introduction to Intel OpenVINO MKL-DNN Plug-in Inference engine Model Optimizer IRTrain a model CPU Deep Learning Deployment Toolkit Deploy Pre-Trained Model Tensorflow, Caffe, MXNet, ONNX
- 21. @IntelSoftware @IntelGraphics 21 Optimization : Cubic Hermite Spline [1/2] ! = 0 ! =1 $% $& '% '& $ ! = 2!) − 3!, + 1 $% + !) − 2!, + ! '% + −2!) + 3!, $& + (!) − !,)'&
- 22. @IntelSoftware @IntelGraphics 22 Optimization : Cubic Hermite Spline [2/2] ! " = 2"% − 3"( + 1 !+ + "% − 2"( + " ,+ + −2"% + 3"( !- + ("% − "(),- ! " = "% "( " 1 2 −2 1 1 −3 3 −2 −1 0 0 1 0 1 0 0 0 !+ !- ,+ ,- Matrix calculations can be accelerated by Intel Openvino . in matrix form
- 23. @IntelSoftware @IntelGraphics 23 Optimization : Batch inference Serial computation Parallel computation
- 24. @IntelSoftware @IntelGraphics IK Solver 24 Optimization for multiple characters [1/2] Character Character Character Character Character Motion timer updates frame number. Trajectory Network Pose Network Next frame motion data Batch Manager Request next frame motion data Next motion frame
- 25. @IntelSoftware @IntelGraphics 25 Frequency of used batch size (10, 14.5 ) (20, 15.3 ) (30, 9.0 ) (50, 7.7 ) (70, 5.5 ) 0 2 4 6 8 10 12 14 16 18 1 3 5 7 10 12 15 20 25 29 40 50 54 70 90 110 130 150 170 190 210 230 250 270 300 370 400 520 540 1000 Frequencyofuse Used Batch Size
- 26. @IntelSoftware @IntelGraphics 26 Finding optimal batch size that achieves maximum throughput while maintaining limited latency Set bs to initial batch size Repeat until we obtain a reliable max latency Start measuring latency If number of inference > bs infer multiple times Else If number of inference <= bs infer once and ignore unused space Finish measuring latency Update max latency End 0 2 4 6 8 10 12 14 16 18 5 10 15 20 25 30 35 40 45 50 Batch size
- 28. @IntelSoftware @IntelGraphics 28 Quality Comparison Our ApproachNumerical Approach
- 31. @IntelSoftware @IntelGraphics 31 Deep Learning based Inverse Kinematics § Character animation becomes more realistic by solving IK. § Conventional approaches for full-body IK are complex or computationally expensive. § Solving full-body IK using Intel Openvino is cost efficient and also flexible. § Process of “Climbing IK Solver” 1. Trajectory network 2. Curve fitting 3. Pose network § Gathered training data from manually crafted reference motion.
- 32. @IntelSoftware @IntelGraphics 32 Optimization using Intel® OpenVINO™ toolkit § CPU is suitable for computing inference on Game-Client. § Deep Learning Deployment Toolkit is high performance framework for computing inference on a CPU. § Model Optimizer converts and optimizes the trained model to IR. On Game-Client, Inference Engine loads this IR and it can now run inference on CPU using optimized hardware plugin (MKL-DNN plugin). § Improved curve fitting performance by computing matrix form of cubic Hermit spline with Inference Engine. § Batching inference task improves performance by parallelizing inference task by the Inference Engine. § Finding optimal batch size to meet limited latency