Apple is Adding Support for Vision Pro’s Input System to WebXR

Apple is adding support for Vision Pro’s unique input system to WebXR, the web standard which allows XR experiences to run right from a web browser.

One of the most unique things about Apple Vision Pro is its input system which eschews motion controllers in favor of a ‘look and pinch’ system which combines eye-tracking with a pinch gesture. On the whole it’s a really useful way to navigate the headset, but because it works so differently than motion controllers, it doesn’t play too well with  WebXR.

But Apple is working to fix that. This week the company announced the latest version of VisionOS (1.1) includes a new input mode for Safari’s WebXR capabilities called ‘transient-pointer’. This new mode provides inputs from the headset in a standardized way which developers can use to understand what users are selecting inside of a WebXR session running on Vision Pro.

Up to this point, WebXR apps typically expect a headset report a continuously updated position of each controller. But Apple says it built Vision Pro’s input system to reveal as little information about the user as possible, so it doesn’t report the pose or position of the user’s hands by default. Instead, it only reveals such information at the moment of the user’s pinch (though it’s possible for a WebXR app to ask for full hand tracking info).

With the new transient-pointer option, when a user pinches the WebXR app will be able to see a ray representing the direction of the user’s gaze and the coordinate position of their pinch. Like in VisionOS itself, the app thus looks at the pinch to decide ‘when’ a user is making an input, and looks at the ray to decide ‘where’ they’re making the input.

For the duration of the pinch, the position of the pinch itself is continuously updated, allowing for interactions like dragging, pushing, and pulling objects. But when the pinch is released, the app no longer has access to the direction the user is looking or where their hand is located.

With these new capabilities, WebXR apps will be able to adapt their interactions to work correctly with Vision Pro.

However, WebXR on Vision Pro is still experimental. Developers must manually enable WebXR capabilities by accessing advanced settings of Safari in the headset. Developers can also experiment with WebXR and the transient-pointer mode using the VisionOS simulator.

The transient-pointer mode for Vision Pro is being baked into the WebXR standard, and has been added to the most recent draft version of the specification. That means that devices which adopt the same input mode will be able to tap into the same WebXR capabilities.