There’s one thing that all mixed reality (and some virtual reality) headsets have in common but not one has truly perfected yet, and that’s passthrough. Through the meticulous arrangement of cameras and lenses, headsets like the Meta Quest Pro, Apple Vision Pro, and even the $299 Quest 2 can simulate your physical surroundings — to varying degrees — as they cover your actual view.
Also: Apple’s Vision Pro labs are drawing audible gasps from developers, says company
Passthrough empowers augmented reality applications like home design, fashion, and interactive learning, use cases that involve some degree of real-world spatial awareness, and a whole lot of computational processing for image simulation.
Naturally, achieving a visual experience similar to how you see things in real life is not an easy feat, especially when you take into account the distance between where the headsets’ cameras are set and where your own “cameras” (or eyes) are positioned. But Meta has a secret weapon stashed within its Reality Labs’ Display Systems Research (DSR) team that it believes has cracked the passthrough code.
Meet Flamera, a pre-production mixed-reality headset that uses “Light Field Passthrough” as a means to accurately replicate your surroundings. By assembling a grid of apertures between the cameras’ lens arrays and sensors, the most optimal light capture is achieved no matter how you move your eyes, and there’s a natural recording of both the distance and motion of the things around you.
A pre-production unit of Flamera. Meta
This aperture module results in what can only be described as a bug-eye appearance, with holes pointing at all directions and angles. The headset takes the raw sensor data captured by all of those holes and pieces it together through depth-dependent reconstruction, according to Meta. You can see a comparison between passthrough mode and reality below.
“The Flamera optical design works best when the headset is thin, which lets us put the passthrough cameras as close to the user’s eyes as possible,” says Grace Kuo, a Research Scientist in Meta’s DSR team. “With our work on light field passthrough, we’re focused on previewing the experience of perspective-correct MR passthrough,” adds Douglas Lanman, Director of DSR.
Just last month, the Flamera was demoed at Siggraph 2023, an annual convention that showcases innovators in the computer graphics and interactive techniques space. “It was definitely among the best quality [augmented reality] passthrough I’ve ever seen,” Anshel Sag, an attendee and Principal Analyst at Moors Insights & Strategy told IEEE Spectrum.
Also: Meta Quest 3 unveiled: Price, features, and when you can buy
Sag also called out the smaller field of view of the beta headset, which can pose a problem for VR/AR applications that require you to look from side to side.
Still, Flamera may be the start of something big. And the slate of new mixed reality headset releases in the coming months, culminating in Apple’s Vision Pro launch early next year, makes technology developed by Meta’s DSR team something to keep an eye on.
