Disney's AR Projection Tech Morphs Actors into On-Screen Monsters

The Disney Research Lab is using projector-based illumination to paint actors' faces during live performances. You know what that means? Disney just took Halloween to a whole new level.

Here's how they did it: The actors move on stage, sans makeup, while the AR light follows them to give them one monstrous appearance after another. The technology is called Makeup Lamps, which is fitting for the way the actors appear — as if they are covered in a thick cake of clown makeup. Scary. Scary. Clown makeup.

The whole process is done in real-time, but not without its challenges. The light is projected according to each actor's specific pose and according to the Research Team's blog, the key challenge with live augmentation is latency:

[A]n image is generated according to a specific pose, but is displayed on a different facial configuration by the time it is projected. Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection. Using infrared illumination, an optically and computationally aligned high-speed camera detects facial orientation as well as expression.

The Disney Research team's high-speed camera is able to keep up with the motions of the actors, reducing delays and misplaced augmented makeup. This means that the actor's augmented face and actual face will never be out of sync.

According to Engadget, the team's solution to the latency issue is called "Kalman filtering." The name refers to an algorithm that uses "measurements over time to make predictions and minor adjustments" to the actors' faces.

With this augmented reality technology, the actors have the freedom to look however they please. If an actor needs to look older, Makeup Lamps will augment wrinkles. If the actor needs to look more colorful, the light system will then be adjusted to meet the demand of the character.

In contrast to existing methods, the presented system is the first method which fully supports dynamic facial projection mapping without the requirement of any physical tracking markers and incorporates facial expressions.

The Research Team's use of Kalman filtering is the first of its kind, although similar facial projection mapping techniques have been attempted before. Last year, Lady Gaga performed at the Grammys where virtual drops of red paint fell down her face to signify David Bowie's iconic lightning bolt.

Next Reality reached out to the Disney Reseach team for comment on the use of this AR technology for a possible future Disney theme park attraction, but unfortunately, the work is not being planned for any exciting amusement park rides as of yet:

This work is all being done for research and there are no planned applications for this work

What do you think Disney Research will think of next? A whole augmented reality themed park? Let your imagination run wild in the comments below!

• Follow NextReality on Facebook, Twitter, and YouTube
• Follow WonderHowTo on Facebook, Twitter, Pinterest, and Google+

Cover image via Disney Research Hub/YouTube