Meta’s Wearable Devices Access Toolkit, coming later this year, will let smartphone apps interact with the company’s smart glasses.
With the initial release of the SDK, Meta says developers will be able to access the camera, speakers, and microphone array of its full glasses lineup: Ray-Ban Meta, Oakley Meta HSTN, Oakley Meta Vanguard, and Meta Ray-Ban Display. Users will have to give each app permission to access their glasses.
Developers could, for example, leverage it to add first-person livestreaming or recording features to their apps. Or they could feed the camera imagery to a mutimodal AI model to analyze what you’re looking at.
At launch, Meta Wearable Devices Access Toolkit won’t support Meta AI. That’s not to say that developers won’t be able to build AI experiences. But they will have to initiate a continuous camera and/or audio stream to and from the glasses to their own AI model of choice, at their own cost, and this will significantly impact battery life. Meta says adding Meta AI integration is a “key area” it’s exploring for future updates to the toolkit.
The SDK also won’t support sending imagery to the Meta Ray-Ban Display HUD, nor accessing gestures from its Meta Neural Band, but Meta says it’s starting to think about how this could eventually work.
Early Developer Experiments
Meta provided an early version of the Wearable Devices Access Toolkit to a handful of developers several months ago, including Twitch, Microsoft, and Logitech Streamlabs.
Twitch and Logitech Streamlabs are using the SDK to let you livestream your first-person view on their platforms, just as you already can on Instagram, while Microsoft is using it for their Seeing AI platform that helps blind people navigate and interact with the world around.
0:00
How 18Birdies is using Meta Wearable Devices Access Toolkit.
One particularly interesting use case comes from 18Birdies. The golf app is experimenting with using Meta Wearable Devices Access Toolkit for real-time yardages and club recommendations, helping golfers without requiring them to take their phone out of their pocket.
