SmartStream

Apple's Worldwide Developer's Conference (WWDC) kicked off with a preview of the long-awaited Vision Pro, an AR (augmented reality) headset featuring micro-OLED display technology that has the potential of changing the way we experience entertainment in the future.

In a video played during the WWDC keynote, office workers donning Vision Pro headsets stood before floating Mac "desktops" opening apps by merely looking at them or using a thumb and forefinger to select, drag, and drop content or resize it into larger-than-life AR windows.

The genius of such “spatial computing” is made possible by an M2 chip (like the one used in Mac computers) working in tandem with a new "R1" chip designed to process info from a "revolutionary new input system" with 12 cameras, five sensors, and six microphones. The goal: to present a lag-free view of the world surrounding the viewer. Everything — including the user’s voice plus eye, face, and hand movements — is monitored constantly in real time.

Unlike other immersive virtual reality (VR) headsets, Vision Pro uses a robust AR interface to overlay virtual elements of your choosing onto the real world and automatically turns transparent to reveal your eyes when it detects another person in the space — a move intended to reinforce the personal connection.

Apple CEO Tim Cook crystalized the headset this way: “Vision Pro is a new kind of computer that augments reality by seamlessly blending the real world with the digital world…You can see, hear, and interact with digital content just like it’s in your physical space. And you control Vision Pro using the most natural and intuitive tools: your eyes, hands, and voice. With Vision Pro you’re no longer limited by a display. Your surroundings become an infinite canvass. Use your apps anywhere and make them any size you want. Capture photos and videos and relive your most important memories in an entirely new way. Watch your movies, shows, and sports and immerse yourself in games on a giant screen.”

A feature Apple calls SharePlay lets you view photos, listen to music, and watch movies with another Vision Pro wearer, creating a lifelike three-dimensional experience — especially with movies — that transcends two people lounging on a sofa in front of a TV. The headset can display photos from your library in a dramatic panoramic format that makes feel as if you are standing in the middle of the location where the photo was taken.

Speakers located close to each ear create a personal spatial audio experience that adjusts automatically to the acoustics of the listening space; sophisticated processing uses audio ray tracing to analyze the surroundings and then compensates for the impact walls, windows, furniture, curtains, etc. have on the sound. For a more fully immersive sonic experience, you can pair Vision Pro with a set of AirPod Pro earbuds or AirPod Max headphones.

Perhaps most compelling to S&V readers, the Vision Pro is capable of presenting a massive 4K virtual screen that you can “put” anywhere thanks to the use of two micro-OLED displays that deliver 23 million pixels to each eye. Watching a movie has been described as hyper-realistic — almost like watching from the inside. The idea is you can replace a bulky, traditional home theater setup with a virtual theater you can take with you wherever you go. Imagine sitting in bed or a camper and feeling like you're in a movie theater.

But there’s a drawback to this idyllic scenario, something that Apple will no doubt continue to work on in the months leading up to the product’s official launch next year: battery time is a paltry 2 hours, meaning you will want to be plugged in when watching a movie.

During the keynote, Disney CEO Bob Iger announced a partnership with Apple that will make an enhanced Disney+ streaming experience available to Vision Pro users on day one of the launch. Enhancements will include the ability to present multiple camera angles on additional screens, real-time sports analysis, and stats that are relevant to the program you are watching. For example, details on characters, filming locations, etc. will be available to viewers of Star Wars and The Mandalorian. National Geographic shows will surround viewers with nature videos. You will even be able to interact with Mickey Mouse as he bounces around your room.

When it comes to its popular FaceTime app, Apple takes a futuristic approach, creating a realistic AI-generated head and shoulders avatar that uses the headset’s built-in cameras to mimic your facial expressions and movements. Some (myself included) see this as a bit creepy, as the idea behind FaceTime has always been to see the real people on the other end of the call.

Apple expects to ship the Vision Pro in 2024 at a price that even some early adopters might find high: $3,500. But Mike Rockwell, head of the AR/VR team, prefaced the price reveal like this: "If you purchased a new state-of-the-art TV, surround-sound system, powerful computer with multiple high-definition displays, and a high-end camera, you still would not have come close to what Vision Pro delivers."

Will Vision Pro have the wherewithal to replace your computer and home theater? That may depend on whether you can envision (or accept) gathering in the living room with your family to watch a movie with everyone wearing headsets.

tvOS 17 Adds Continuity Camera for Expanded Features
Apple also announced a tvOS 17 update intended to facilitate more robust interactivity between iPhones, iPads, and Apple TVs. Beyond being able to “AirPlay” content from an iPhone or iPad, the upgrade introduces Continuity Camera, which wirelessly connects the Apple TV to your device's camera and microphone in real time, making it possible to use FaceTime calling on Apple TV.

Continuity Camera also makes it possible to use video conferencing apps like Webex by Cisco and Zoom. Both will launch on tvOS later this year.

With the tvOS17 update, you will also be able to start calls on an iPhone or iPad for hand off to an Apple TV or initiate calls directly from the Apple TV. And FaceTime on Apple TV will employ Apple’s Center Stage feature, which uses machine learning to keep everyone in the video frame, even as they move around the room, with the ability to use simple hand gestures to generate onscreen effects like a river of hearts or fireworks.

SharePlay, which works with FaceTime on an iPhone or iPad, is coming to Apple TV, too. The feature will include a side-by-side view so you can watch a movie or look at photos on one side of the screen while seeing the reaction of the FaceTime caller on the other side.

Another feature heading to Apple TV is the Apple Music Sing feature introduced on iPhones last year. Thanks to Continuity Camera, users will be able to see themselves on the TV screen and add entertaining filters while singing along to their favorite songs.

Aside from the new Continuity Camera features, Apple has made other helpful updates. When the newest Apple TV 4K was released, many complained that you couldn't find the remote in the Find My app like you could with other Apple devices or AirTags. To rectify the situation, Apple is adding a remote finder to the iPhone that uses a blue dot, which gets bigger as you move closer to the missing remote.

Other tvOS17 improvements include:

• Curated photo memories to use as a screen saver
• Enhanced dialogue when an Apple TV 4K is paired with a second-generation HomePod; the feature separates dialogue from sound effects and other audio to improve intelligibility
• Dolby Vision 8.1 Support
• Apple Fitness Plus for setting a workout or meditation schedule with the ability to create back-to-back workouts and meditations and lower the trainer’s voice while cranking up background music (via a feature called Audio Focus)

AirPods Pro and AirPods Max Get Software Updates
In another announcement, Apple introduced software updates that bring new automated features to AirPods Pro earbuds and AirPods Max headphones.

Adaptive Audio automatically adjusts transparency and active noise cancellation based on the noise level in the surrounding environment. Examples include tailoring active noise control (ANC) to let in essential sounds like an approaching bicycle if you’re on a trail or notching up the ANC to quiet the noisy washing machine in the next room over.

Conversation Awareness lowers the volume of your music or podcast, while enhancing voices in front of you and reducing background noise when someone is speaking to you.

Personalized Volume uses machine learning to automatically fine-tune media and listening preferences based on settings you have previously used in different environments.

Automatic Switching eliminates the frustration of having to manually switch AirPods from one Apple device to another. For example, you can make a call on your iPhone and quickly switch back to watching YouTube video on your Mac.

iPhone AirPlay Gets Smarter
In iOS 17, AirPlay will use machine learning to predict where to play certain kinds of media — on the living room Apple TV or office Mac, and so forth.

AirPlay is also being added to televisions in some hotels, allowing Apple users to watch Netflix or Apple TV Plus content without having to sign in their account. AirPlay will be available before the end of the year, starting with IHG Hotels and Resorts.

Welcome to the Future
The announcement of the Vision Pro along with the many new features are substantial innovations that could alter how we use our iPhones for entertainment and communication, how we wear our AirPods, and what home entertainment will look like a few years down the road. Changes may not be immediate, but in the future, when you sit in front of your TV for a Zoom call, you will forget about the days when you had to strain to see all of the meeting’s participants on your phone or laptop.