Apple's Open-Source On-Device AI Instantly Turns Images Into Volumetric Scenes

Apple’s open-source on-device AI model instantly turns images into scenes, and Vision Pro owners can try it out in the app Splat Studio.

Since visionOS 26, Apple’s own Photos app has included a one-click feature to almost instantly turn any image into a ‘Spatial Scene’. It’s essentially a volumetric photo with a limited area of viewing freedom, which you can slightly lean around to “peak” into.

Meanwhile, over the past year or so multiple open-source and proprietary AI systems emerged that can go much further, turning a photo into a scene that you can freely explore, even walk around. For example, Marble lets you do this in your headset’s web browser and explore the scene in WebXR.

Marble Turns An Image Into A WebXR Volumetric Scene In Minutes

Marble, an AI model from World Labs, can turn a single image into a volumetric scene that you can view in WebXR in a matter of minutes.

UploadVRDavid Heaney

Marble is a computationally expensive server-side model, however, that takes minutes to produce its result. And that’s what makes Apple’s SHARP particularly interesting.

SHARP runs on typical consumer devices, with general CPU support as well as Nvidia CUDA and Apple Silicon Metal hardware acceleration, taking less than a second to complete on most hardware.

In a rare move from Apple, SHARP is free and open-source, with the code available on GitHub. You can easily download and run it on a Mac, for example.

As with almost all of the remarkable advancements in 3D reconstruction over the past few years, it generates a Gaussian splat, fitting millions of semitransparent colored blobs (Gaussians) in 3D space so that arbitrary viewpoints can be rendered realistically in real-time. You receive the result as a .ply file that can be rendered in any standard 3DGS viewer.

For Apple Vision Pro owners, Portugal-based developer Rob Matwiejczyk built a visionOS app that integrates Apple’s SHARP model into an easy-to-use graphical interface and eliminates the need to use a Mac or PC.

Called Splat Studio, the app is available for free on the App Store, and runs entirely on-device. Just choose any image from your Photos library and it instantly gets turned into a 3D scene floating in front of you, which you can rotate, move, and scale with your hands.

I tested Splat Studio on the M5 Apple Vision Pro, using the same Steam Dev Days 2014 VR room I used to test Marble. For comparison, I also turned the same image into a Spatial Scene in the visionOS 26 Photos app. You can see footage of the Splat Studio result above, and of the Spatial Scene below.

The Splat Studio app turned the image into a scene in around 20 seconds, compared to the near-instant result of Apple’s Photos app, but it’s unclear how much of this is truly due to the SHARP model compared to any overhead the Splat Studio app may add.

As for the result, while the Apple Photos Spatial Scene lets you peer into the scene, the degree to which you can move in each direction is relatively limited. Meanwhile, the SHARP result in Splat Studio lets you freely move around the scene. The tradeoff, as with many generative AI results, is some detail loss, as well as hallucinated details the further you go from the original perspective of the image.