The Quest 3 & Quest 3S “Passthrough Camera API” can now be used in shipping apps on the Meta Horizon Store.
The capability was made available to all developers as an experimental release six weeks ago, meaning they could experiment with it, and in theory distribute APKs with it (on SideQuest, for example), but couldn’t include it in Meta Horizon Store apps until now.
With v76 of the Meta XR Core SDK the capability is no longer considered experimental, and thus can be included in shipping apps “after a review process”.
What Is Passthrough Camera Access?
While headsets like Quest 3 use cameras to let you see the real world, until recently only the system software had raw access to these cameras. Third-party developers could use passthrough as a background, sure, but they didn’t actually get access to it. They instead got higher-level data derived by the system, such as hand and body skeletal coordinates, a 3D mesh of your environment with bounding boxes for furniture, and limited object tracking capabilities. That meant they couldn’t run their own computer vision models, which severely limited the augmentation capabilities of these headsets.
For the “Passthrough Camera API” to work, you as the user need to grant the app permission to access your headset cameras, just as you would the microphone. If granted, the app gets access to the forward-facing color cameras, including metadata like the lens intrinsics and headset pose, which it can leverage to run custom computer vision models.
Examples of how apps can use this include scanning and tracking QR codes, detecting a game board on a table to add virtual characters and objects to it, detecting physical objects for enterprise guide experiences, and integrating the visual AI functionality of cloud-hosted large language models (LLMs). Developers are only limited by which real-time computer vision models can run on the XR2 Gen 2 chipset performantly, or which cloud-hosted image analysis models they’re willing to pay for.
Meta software engineer Roberto Coviello’s QuestCameraKit samples.
The passthrough camera stream is provided to the app with up to 1280×960 resolution per camera at 30FPS, with a stated latency of 40-60 milliseconds. That means it isn’t suitable for tracking fast moving objects, such as custom controllers, nor for discerning fine features like small text.
Technically, at a base level, there actually is no specific Meta Quest Camera Passthrough API. Developers do need to request a Horizon OS specific Headset Cameras permission, but otherwise Quest’s passthrough camera access leverages Android’s existing Camera2 API, extending it to return the headset pose via OpenXR, and the Camera2 API is what developers of custom engines, or source code for Unreal or Godot, use for it. This also means the same code should work on Google’s upcoming Android XR platform, set to debut in Samsung’s standalone headset, with only the permission request being different.
For Unity, developers can easily access the cameras through Unity’s WebCamTexture API, which is how they already access phone, tablet, and PC cameras and webcams in the engine. A limitation here, however, is that Unity’s WebCamTexture API only supports one camera at a time, not both.
Walkthrough from Meta software engineer Roberto Coviello.
Interested developers can find Quest passthrough camera access documentation here: Unity / Native Android.
Meta has published five official Unity samples on GitHub: CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDectection, ShaderSample. Meta software engineer Roberto Coviello has separately published QuestCameraKit on GitHub, a collection of five further samples: Color Picker, Object Detection, QR Code Tracking, Frosted Glass Shader, and OpenAI Vision Model.
