Header Banner
nextreality.mark.png
Apple Snap AR Business Google Instagram | Facebook NFT HoloLens Magic Leap Hands-On Smartphone AR The Future of AR Next Reality 30 AR Glossary ARKit Dev 101 What Is AR? Mixed Reality HoloLens Dev 101 Augmented Reality Hololens How-Tos HoloLens v. Magic Leap v. Meta 2 VR v. AR v. MR

How Face Tracking & Augmentation Technology Works

3D wireframe model of a human head, showing front and side views.

Thanks to Snapchat, we're all familiar with face swapping and augmentation thanks to some clever, fun technology. But that's just the beginning, because this kind of augmented reality can do so much more.

We spoke to Peter Busch, VP of Business Development at Faceware Technologies and Image Metrics. You can check out the full interview below, where he explains how face tracking works and offers a variety of useful applications.

We've already seen how filters can preview cosmetics and even diagnose skin conditions, but what makes that possible goes beyond detecting a basic facial structure. Most face-tracking software is based upon the Viola–Jones object detection framework, which provides an algorithm for separating faces from non-faces. Building upon that "simple" detection step, the software can begin to recognize the components of the face to build a general map for effects.

Facial tracking technology demonstration with expressions and features.

The basics of Image Metrics' technology.

That "map" is more of a 2D mesh structured around the shape of your face, looking for shapes and differences. By also looking at the contrast of light in the live image, the software will quickly find the most expected parts—your eyes, nose, and mouth—and mark those accordingly. With that data, your face is both recognized by the software and able to be tracked and used as a basis for effects.

Three individuals with creative face paint designs showcasing various themes.

Diehard Fan app lets you apply "face paint" without getting messy.

The Viola-Jones algorithm isn't new. It was proposed in 2001. Only recently has mobile hardware become powerful enough to handle this type of augmented reality in real-time. In fact, Busch notes that face-tracking and augmentation technology can do a lot more than we see in Snapchat and similar apps. Currently, our smartphones limit the degree of realism and detail we can achieve.

We're actually having to clamp down the tech, and kind of handcuff it a little bit because of the nature of the device. We're actually capturing a lot more information in real time than the devices right now ... are able to do. So with every new generation of phone that comes out ... we can kind of unlock those handcuffs a little bit and start to do more and more and more.

With quickly-evolving hardware, this facial augmentation will only become better. Image Metrics already offers some pretty detailed effects, and their head tracking keeps those effects in place even when the subject's head moves from side to side. They also analyze the texture of the subject's face, which has applications in the beauty and health industries and can help create more realistic renderings of personalized 3D avatars.

Where this technology will take us remains to be seen, but as hardware improves so do the possibilities that creatives have to make new, interesting experiences through tracking and augmenting faces.

Cover image by Kluva/Shutterstock

You already know how to use your phone. With Gadget Hacks' newsletter, we'll show you how to master it. Each week, we explore features, hidden tools, and advanced settings that give you more control over iOS and Android than most users even know exists.

Sign up for Gadget Hacks Weekly and start unlocking your phone's full potential.

Related Articles

Comments

No Comments Exist

Be the first, drop a comment!