Apple’s Vision Pro is set to gain significant AI capabilities with the upcoming VisionOS 27 release, demonstrated at WWDC. This update focuses on integrating AI directly into the headset’s functionality, specifically through a feature called Visual Intelligence.
What Happened
VisionOS 27, currently available as a developer beta, introduces AI-powered features centered around Siri. The key addition is the ability for Siri to visually understand the user’s surroundings. Unlike Google’s Gemini Live, Apple's implementation doesn’t offer continuous scene analysis; instead, Siri captures a snapshot of the user's field of view based on gaze tracking. This allows users to ask Siri about objects within their environment. Other additions include the ability to convert personal panoramic photos into 3D immersive backgrounds and preview 3D objects from Mac apps directly within the Vision Pro environment. However, Apple continues to rely on Macs as primary companions for the Vision Pro, with no new core apps developed for the headset, nor any official support for the Apple Watch, iPhone, or iPad as connected accessories.
Why It Matters
This update is significant because it represents Apple’s commitment to spatial computing and its integration of AI. The Visual Intelligence features demonstrate how Apple envisions future AR glasses functioning—leveraging on-device AI to understand and interact with the real world. Currently, the Samsung Galaxy XR headset offers similar camera-aware AI capabilities, but Apple’s approach with gaze-triggered snapshots presents a different trade-off between privacy and continuous awareness. For developers, VisionOS 27 offers new APIs to tap into these AI capabilities, allowing them to build more contextually aware applications. The continued reliance on the Mac as a companion device, however, may limit the Vision Pro’s appeal for users seeking a completely standalone experience. The lack of new core apps and accessory support suggests Apple is still defining the ecosystem around Vision Pro.
What To Watch
It remains to be seen how developers will utilize the Visual Intelligence features in VisionOS 27. Testing the developer beta will reveal the practical applications and limitations of Siri's object recognition. It will also be important to observe whether Apple addresses the lack of core apps and accessory support in future updates. The success of Vision Pro, and its potential to influence the future of AR/VR, hinges on Apple’s ability to build a compelling and integrated ecosystem.