SmartStream

Apple previewed a number of compelling new features and capabilities at its annual Worldwide Developer’s Conference (WWDC), including some of the most significant changes we’ve seen in years with Apple’s version of AI figuring prominently into 2024 updates and features. With our usual focus on all things audio and video, here’s a summary of the key announcements.

Apple TV. The new tvOS18 version of the Apple TV operating system will bring improvements in audio and video, including support for ultra-widescreen 21:9 aspect ratio images, better sound quality, and a new addition to the Apple TV app called InSight that provides real-time access to detailed information about the Apple TV+ show you’re watching.

Similar to Amazon’s X-ray feature, InSight creates an onscreen display showing the names and faces of actors in the scene you’re watching as well as the song and artist playing at the time. Clicking on the name of an actor brings up a complete filmography and additional background. When using an iPhone as an Apple TV remote, InSight also appears on the phone, so there’s no need to hit the pause button to get background info. The feature also makes it easy to add the song playing to an Apple Music playlist.

The tvOS18 update will also bring new refinements to the Enhance Dialogue feature on Apple TV 4K, which will leverage AI-powered machine learning and computational audio to improve clarity by better isolating voices from background music and other competing audio elements, which is especially important in action-packed movies. Apple says the feature works with its AirPod earbuds and HomePod smart speaker as well as with HDMI-connected speakers (including those built into a TV) and other Bluetooth devices.

In a related update that adds a new measure of convenience, subtitles will automatically appear when the movie language doesn’t match the language on the device or when the device is muted or in skip mode.

Finally, Apple will expand the selection of screensavers that can be displayed when your Apple TV is idle to include portraits, scenes from TV shows and movies, a whimsical Snoopy and Woodstock animation, portraits, and favorite photos.

HomePod. Apple’s updates for 2024 will make it easier for anyone to control music playing on a HomePod or HomePod mini, even if they don’t have a subscription to Apple Music. Using Apple Music’s SharePlay feature, a house guest will be able to simply movie his or her iPhone close to your iPhone — or scan the QR code in the SharePlay window on another iPhone — to control playback and add songs to the playlist.

Owners of a HomePod (or other Spatial Audio-compatible speaker or device) will also enjoy Spatial Audio with support for Dolby Atmos when using AirPlay to stream music from an iPhone or iPad.

AirPods Pro.
Apple announced a few upgrades for its AirPods Pro wireless earbuds, including the ability to address Siri with head gestures — ideal for those times when you’re in a crowded environment and don’t want to call attention to yourself by saying commands aloud. With the new Siri Interactions, you won’t have to talk. Instead, you will be able to answer or reject a call by gently nodding or shaking your head. The yes/no gestures can also be used to interact with messages, manage notifications, and more.

Gamers will be happy to learn that Apple also plans to “significantly reduce audio latency” and expand head-tracking Spatial Audio beyond movies and music to include gaming. For multiplayer gaming, audio will be elevated to 16-bit/48-kHz quality to boost clarity when gamers are chatting with one another.

Apple will also use machine learning to improve voice isolation when AirPod Pro users make a call. The update promises to enhance voice quality while applying noise cancellation to remove distracting background noise such as wind and city traffic.

Vision Pro. Apple’s Vision Pro mixed-reality headset has only been available for a few months, yet it already supports 2,000 spatial (immersive) apps and 1.5 million iOS- and iPad OS-compatible apps. The headset will also get an updated operating system, visionOS 2, which will include new hand gestures to make navigation easier and add new features, including a few that will be of interest to S&V readers.

Topping the list is the option to view an ultra-wide 8K virtual display that is equivalent of two 4K monitors side by side and “spatial video” processing that will make possible a 180° viewing experience. The update will also bring Multiview for live sports to Vision Pro through the Apple TV app, allowing super fans to stream up to five games simultaneously, in addition to supporting AirPlay from an iPhone, iPad, or Mac.

In other updates, mouse support will be expanded and Apple’s Magic Keyboard will remain visible, even when users are fully immersed in an environment or app, while the new Systemwide Live Captions feature adds subtitles to live conversations and audio from apps to aid users who are deaf or hard of hearing.

The visionOS 2 update will also make it possible to create “spatial photos” from images already in your library by using advanced machine learning to convert 2D photos into 3D by creating a pair of separate images for the left and right eyes. At the heart of the technology is a technique used since the early days of photography when a stereoscope was used to merge left- and right-eye views of the same scene so they could be viewed as a single three-dimensional image. Think of the process as a sophisticated version of a 1950’s View-Master with its Kodachrome “wheel” of stereoscopic pictures. Vision Pro Users will be able to share spatial photos through SharePlay in the Photos app.

For those who still use a digital camera to take photos, Canon will release a new spatial lens for its EOS R7 camera that records separate right and left images, enabling it to shoot both spatial photos and video. Serious users will be able to edit spatial videos on a Mac and create immersive titles and effects using an updated version of Final Cut Pro. Creators will also be able to share the videos on the Vision Pro Vimeo app.

Apple Intelligence. As you have probably already guessed, AI and advanced machine learning are coming to the Apple ecosystem in the form of Apple Intelligence. A key difference between Apple’s AI and other implementations of AI is that it will be deeply integrated into the new iOS 18, iPadOS 18, and MacOS Sequoia operating systems, meaning most of the processing will be done on the device as opposed to in the cloud, which Apple says will ensure a level of privacy that is not possible with cloud-based AI.

This also means that it is unlikely that earlier Apple devices will be able to handle Apple’s new AI system because advanced processors will be needed to understand language as well as write and create images using personal context across apps without going online. You will be able to ask complicated questions such as: “Will I have enough time to make it to my meeting after the earlier meeting?” Siri will be able to respond because it knows the length of the meetings, the start time of the second meeting, the location of both (if it was in the calendar or listed in an email) and will be able to predict how long it will take to get from one location to the other, while factoring in traffic.

In theory, Siri should be able to do a better job of answering your questions by using context across apps and evaluating the content of your emails, texts, etc. If a query goes beyond the scope of what is on your device, you will, of course, have the option of searching for answers online. Apple has also partnered with ChatGPT, which will be available to Apple users without a subscription.

In addition to understanding natural language and being able to rely on a number of apps to obtain answers, Apple Intelligence will make it possible to:

• Transform a crude sketch into a detailed, polished illustration to go with your notes or a document.

• Automatically create a summary describing the contents of an email instead of showing its first two sentences in the preview.

• Automatically organize email into categories such as shopping, promotions, and priority.

• Create a summary or table of contents to make it easier to find the information you need when visiting a webpage in Safari.

• Rewrite any documents, emails, texts, etc.

• Display a specific picture in Photos based on a verbal description such as, “Show me the photo of Jane wearing a red coat on the beach at sunset.”

• Use Image Playground to create images by choosing categories of objects you want to put together or by circling an empty space in a document and letting AI create an illustration based on its content.
• Design your own emojis.

And That’s Not All Folks
The 2024 WWDC keynote lasted an hour and 45 minutes and touched on every aspect of Apple’s sprawling business, so suffice it to say there were many more announcements than we have covered here, including those pertaining to new features for iOS18 and MacOS Sequoia. Among them, a new feature called Math Notes, which solves handwritten problems and equations, the ability to fully interact with your iPhone on a Mac, texting over a satellite connection, and much more.

All of the many new features and upgrades heading our way in 2024 promise to significantly enhance the capabilities of all Apple devices, but you will need to check to see which of the new features your model will be able to support. The public beta versions of Apple’s operating system software will be available in July, with final versions launching sometime in the fall.