Facebook puts wearers’ actual 3D eyes on VR headset to make talking to people less ‘jarring’ and ‘unnatural’

Facebook’s Reality Labs, the area of the company that develops augmented and virtual reality products, has built a way for people to see the eyes of someone wearing an AR or VR headset.

The Oculus headsets, which are owned by Facebook, use externally facing cameras to help users see their surroundings while wearing the hardware (which Facebook calls ‘passthrough’). This is the opposite, or ‘reverse passthrough’.

“The idea for this experimental reverse passthrough research occurred to [research scientist Nathan] Matsuda after he spent a day in the office wearing a Quest headset with Passthrough, thinking through how to make mixed reality environments more seamless”, Facebook explained in a blog post.

“Wearing the headset with Passthrough, he could see his colleagues and the room around him just fine. But his colleagues couldn’t see him without an external display. Every time he attempted to speak to someone, they remarked how strange it was that he wasn’t able to make eye contact. So Matsuda posed the question: What if you could see his eyes — would that add something to the social dynamic?”

Matsuda demonstrated this idea first in 2019, using a Rift S headset with a 3D display attached to the front. An image of his face was rendered using an in-game engine, and mimicked where he was looking via a pair of eye-tracking cameras. At the time, FRL Chief Scientist Michael Abrash described it as “goofy” and a “novelty”, but now says the idea has “genuine promise”.

open image in gallery

Facebook says a three-dimensional display is necessary to avoid the eyes being “visually jarring” and “unnatural”, as a 2D flat projection was offset from the user’s actual face; a 3D display lets allows “the user’s eyes and face to appear at the correct position in space on the front of the headset”.

Such an idea is still far from touching a consumer product; the hardware is “tethered by many cables” and Facebook says the eye and facial renderings are not yet lifelike enough. Nevertheless, Facebook believes this could be a beneficial path that will serve different users in different scenarios long-term.

“There will be situations where people will need the true transparent optics of AR glasses, and others where people will prefer the image quality and immersion of VR”, it writes.

Facebook’s ventures into mixed reality will first come to consumers in a collaboration with Ray-Ban, CEO Mark Zuckerberg has said. It is unclear what functionality they will have (Mr Zuckerberg only alluded to “neat features”) but previous reports have suggested that they will be able to take calls and stream video to social media, similar to Snapchat’s Spectacles.

As well as the Ray-Ban collaboration, Facebook is also developing ‘Project Aria’ – smart glasses that would “add a 3D layer of useful, contextually-relevant and meaningful information on top of the physical world” – although the launch of that product has been delayed due to the coronavirus pandemic.

In the long-term, these all point to Facebook pivoting towards the ‘metaverse’, an “embodied internet” that would work across desktops and laptops, headsets, and smartphones to interlace the digital and physical worlds with interoperability across services and a built-in economy.