How Face Tracking & Augmentation Technology Works

Thanks to Snapchat, we're all familiar with face swapping and augmentation thanks to some clever, fun technology. But that's just the beginning, because this kind of augmented reality can do so much more.

We spoke to Peter Busch, VP of Business Development at Faceware Technologies and Image Metrics. You can check out the full interview below, where he explains how face tracking works and offers a variety of useful applications.

We've already seen how filters can preview cosmetics and even diagnose skin conditions, but what makes that possible goes beyond detecting a basic facial structure. Most face-tracking software is based upon the Viola–Jones object detection framework, which provides an algorithm for separating faces from non-faces. Building upon that "simple" detection step, the software can begin to recognize the components of the face to build a general map for effects.

How Face Tracking & Augmentation Technology Works — The basics of Image Metrics' technology. *Image via Image Metrics*

That "map" is more of a 2D mesh structured around the shape of your face, looking for shapes and differences. By also looking at the contrast of light in the live image, the software will quickly find the most expected parts—your eyes, nose, and mouth—and mark those accordingly. With that data, your face is both recognized by the software and able to be tracked and used as a basis for effects.

The Viola-Jones algorithm isn't new. It was proposed in 2001. Only recently has mobile hardware become powerful enough to handle this type of augmented reality in real-time. In fact, Busch notes that face-tracking and augmentation technology can do a lot more than we see in Snapchat and similar apps. Currently, our smartphones limit the degree of realism and detail we can achieve.

We're actually having to clamp down the tech, and kind of handcuff it a little bit because of the nature of the device. We're actually capturing a lot more information in real time than the devices right now ... are able to do. So with every new generation of phone that comes out ... we can kind of unlock those handcuffs a little bit and start to do more and more and more.

— Peter Busch

With quickly-evolving hardware, this facial augmentation will only become better. Image Metrics already offers some pretty detailed effects, and their head tracking keeps those effects in place even when the subject's head moves from side to side. They also analyze the texture of the subject's face, which has applications in the beauty and health industries and can help create more realistic renderings of personalized 3D avatars.

Where this technology will take us remains to be seen, but as hardware improves so do the possibilities that creatives have to make new, interesting experiences through tracking and augmenting faces.