'HeadOn', An AI That Transfers Torso, Head Motion, Face Expression And Eye Gaze

A team of researchers has developed an AI that can transfer a person's real-time facial expressions, eye movements, and poses to a portrait - making it as if the person in the image is actually talking and moving.

Constantly improving, the AI project is called ''HeadOn'. Developed by the same people who created 'Face2Face', HeadOn is like its successor, as the white paper calls it: the "first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze."

The AI combines several technologies, taken from the original work on Face2Face which powers a large portion of HeadOn capabilities.

But the team improved the technology to make it able to transfer the torso as well as the head motion, to go alongside with the original person's facial expressions. In short, HeadOn is like a bulked up and improved Apple's Animoji.

"We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze," said the researchers at Visual Computing Group.

Given a short RGB-D video of the target actor, the team automatically construct a personalized geometry proxy that embeds a parametric head, eye, and also a kinematic torso model. Then, a real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor.

On top of the coarse geometric proxy, the team proposes a video-based rendering technique that composites the modified target portrait video via view- and pose-dependent texturing, and creates photo-realistic imagery of the target actor.

The team then propose to continuously track the face and torso of the source actor, all that in real-time.

"We extensively evaluate our approach and show significant improvements in enabling much greater flexibility in creating realistic reenacted output videos.".

The result of this technology is realistic and can be deceiving. To some, it can also be incredibly creepy.

This may cause some privacy issues as hackers can potentially wreak someone's life by just accessing a photo of them.

But just like other technologies that came before it, HeadOn misuse can be prevented, like for example, if the developers can make the output detectable.

"Even though current facial reenactment results are impressive, they are still fundamentally limited in the type of manipulations they enable. For instance, these approaches are only able to modify facial expressions, whereas the rigid pose of the head, including its orientation, remains unchanged and does not follow the input video. Thus, only subtle changes, such as opening the mouth or adding wrinkles on the forehead are realized."

Initially, HeadOn is far from perfect: it doesn't work well on HD resolutions since the digital artifacts can be clearly seen. The AI also doesn't know how to process a person's image with long hair without producing glitchy results.

Published: 
07/06/2018