DI4D worked closely with Infinity Ward to deliver groundbreaking facial animation for the best-selling video game Call of Duty Modern Warfare II. DI4D had previously delivered 25 minutes of facial animation for the cinematics in Call of Duty: Modern Warfare (2019). This time, DI4D used its new PURE4D solution to process over four hours of facial performance data for both cinematics and in-game animation, making it DI4D’s largest-scale project to date.
4D Capture for Improved Digital Double Facial Rig Creation
In order to obtain the highest possible level of graphical realism, Infinity Ward committed to an animation pipeline in which the in-game characters are digital doubles of the actors who play them. This first required the 3D shape and appearance of all 14 of the main cast to be captured in super high detail using Activision’s Light Stage system. A Light Stage scan of each actor with a neutral facial expression was retopologized and used by Infinity Ward to infer automatically an initial FACS-based animation rig for that actor using its FaceGEN system.
Next, DI4D captured an extensive series of dynamic facial expressions and several phonetic pangrams of each of the main cast with a DI4D PRO system. (Phonetic pangrams are phrases designed to include all 40 basic sounds or phonemes in the English language and hence all facial shapes required to reproduce them.) As the DI4D PRO system comprises nine synchronized, 12-megapixel machine vision cameras, running at 30 frames per second, it is able to capture a very large number of subtle “in-between” dynamic facial expressions.
DI4D processed the machine vision camera data first using photogrammetry to produce a raw 3D scan sequence, in which a separate 3D scan is reconstructed for every frame of the video data. The retopologized neutral expression mesh from the Light Stage was then registered precisely to one frame of the raw 3D scan sequence data in which the actor has a similar expression to the light stage neutral. The registered mesh was then tracked accurately through the whole 3D scan sequence using optical flow to apply a consistent mesh topology to every frame in 3D scan sequence. This results in 4D data that is represented as dense point cache animation.
One of the main advantages of DI4D PRO data is that it captures a large number of subtle co-articulated facial shapes, not just a sparse set of extreme expressions associated with traditional FACS scans. Infinity Ward used the DI4D PRO data to supplement and refine the initial FACS-based rig they created for each actor. This approach to facial rig creation resulted in an exponential time saving compared to a traditional rigging process involving sculpting 100s of facial shapes. It also ensured that the facial rig was able to more accurately represent the full range of the actor’s facial expressions, allowing artists to focus on finer details that would not normally receive any attention.
Driving Digital Doubles with High Fidelity, In-Game Facial Animation
The facial performances that were used in the game were captured at Infinity Ward’s dedicated Performance Capture (PCAP) stage. For every scene, facial performance data was captured using Head Mounted Camera (HMC) systems simultaneously with vocal performance and body motion capture. The simultaneous capture of all of the elements of performance from the same actor is important to achieve accurate synchronization and a high degree of realism.
DI4D processed hundreds of selects made from the HMC footage using its new proprietary PURE4D solution, which is designed to bring digital double facial animation up to a new level of realism. PURE4D combines the actors’ facial performance data captured with HMCs during PCAP with the higher fidelity dynamic expression data captured with the DI4D PRO system. PURE4D also uses advanced machine learning algorithms to reduce significantly the amount of manual clean-up normally associated with processing 4D data, and hence allowed 1,000s of seconds of high quality facial animation data to be delivered in a timely manner.
The type of point-cache animation generated by PURE4D is able to faithfully represent an actor’s facial performance with a very high degree of accuracy but it can be very heavy for in-game use and difficult to manually edit. Infinity Ward addressed this by solving the point-cache animation to their rig’s animation parameters. This process introduced semantic control allowing an animator to modify the performance, giving final creative control to the animation team.
However, because the rig and PURE4D data were both derived from the same set of DI4D PRO data, it was still possible to obtain a rig solve that was objectively very close to the point-cache animation and hence to the actor’s original performance. The rig solve process also allowed the memory footprint of the animation to be scaled down for in-game use on a variety of hardware platforms.
“DI4D technology allowed us to implement incredibly realistic facial animations in a way that saved us a lot of time in the production process. This meant we could focus on other important aspects of the game’s development without sacrificing the level of immersion and detail we were aiming for,” said Sean Comer, Principal Machine Learning Engineer, Infinity Ward.
Call of Duty: Modern Warfare II is available on PlayStation 4 & 5, Windows, Xbox One and Xbox Series X/S (callofduty.com).
To find out more about DI4D facial animation services, visit di4d.com.
