Quest Store & App Lab apps can now use hand tracking and Quest 3 or Quest Pro controllers simultaneously.
This feature, called Multimodal, has been available in the Meta XR SDK as an “experimental” feature since July, meaning developers could play with it or ship it via a third-party route like SideQuest but couldn’t ship it on the official Quest Store and App Lab.
Since the launch of Hand Tracking almost four years ago it has always been possible to build Quest apps that support both hand tracking and controllers, where you pick up or set down your controllers to switch between the two modes at any time. But there’s a delay between switching modes, and of course in hand tracking mode the controllers disappear and stop tracking.
0:00
Footage of Multimodal posted by Meta CTO Andrew Bosworth.
With Multimodal controller tracking instead continues, so there is an instantaneous transition. But more importantly, it enables mixing of hands and controllers. This should allow one-controller games like GOLF+ and Walkabout Minigolf to track your other hand, for example, if the developers of those games implement this feature.
The controllers could theoretically also be attached to objects or body parts to act as generic trackers, if developers want. This would work a lot better with Touch Pro controllers than with Touch Plus though, given they would often be occluded from the headset’s view.
0:00
Footage of Multimodal captured by VR enthusiast Luna.
Multimodal only works with Quest 3 or with Touch Pro controllers, which come with Quest Pro or can be bought for Quest 2 and 3 for $300. It won’t work with Quest 2 controllers. That’s because Quest 3 already constantly runs hand tracking alongside controller tracking to fill in the gaps when a controller is temporarily occluded, while Quest 2’s chipset likely isn’t capable of performantly running both at once. And of course, Touch Pro controllers track themselves, so have no performance requirement.
A developer enabling multimodal also restricts them from using some other features, including Inside-Out Body Tracking, the hand tracking Fast Motion Mode and new Wide Motion Mode, and tracked keyboards.
