SIGNALS

Of course, seeing is no longer believing — especially if you're watching anything on video. You've seen the deepfakes of celebrities and politicians and everyone recognizes the potential societal harm that is brewing here. But, this AI AV cloud has at least one silver lining.

Case in point: The movie Fall is a survivalist thriller telling the fictional story of two adventurous women who become stranded atop a 2,000-foot-tall broadcast tower. When the movie was acquired for distribution by Lionsgate, the studio was concerned that the movie's F-bombs would result in an R rating, thus restricting the movie's box office. The studio asked the filmmakers to tone down the language, for example, changing “f***ing” to “freaking.”

The filmmakers could have cut those scenes, completely neutering the movie. They could have reshot those scenes, resulting in a huge price tag. They could have dubbed over the offending words, resulting in the typically clumsy obscene lip motion and family-friendly audio. But they chose a fourth option. They commissioned Flawless, a company with a technological solution called TrueSync, AI-based software originally designed to more convincingly dub film dialogue into other languages by altering the picture to match the replaced audio.

The movie's actors recorded more family-friendly terminology and TruSync was used to alter the original images, specifically the actor's lips and faces, to match the new vocabulary. In other words, the program performs visual dubbing to match audio dubbing. This extra post-production of “neural reshoots” took about 2 weeks, and the result is pretty impressive. You can view a short demo clip here. You can see and hear the original video and audio, see the facial mapping, dialogue removal, and replacement of image and dialogue.

In addition to the F-bomb repair, the clip shows how a movie can be visually dubbed into a foreign language. This kind of image manipulation can allow more faithful foreign-language dubbing; for example, the translated script doesn't have to be changed (i.e. butchered) to match the mouth movements. The foreign-language voice actor can use an accurate translation, and the video can be modified to fit, rather than vice versa.

Interestingly, this kind of technology is already beginning to change the movie-making process. For example, instead of shooting a scene several times, perhaps with slightly different dialogue, a filmmaker can potentially save money by shooting the scene once, then tweaking it in post production. Is this the beginning of flawless dubbing, and the end of subtitles? Maybe. In any case, hopefully the days of the lips saying one thing, while you're hearing something else, are coming to an end.

And finally, the obligatory lip-sync joke: How can you tell a [insert despised profession here] is lying? Easy. Their lips are moving.