The Israeli startup DeepDub claims that it can dub films, television shows, and video games while retaining the pitch and tone of the original actors in any given process. This allows them to recreate the voices of famous actors in other languages, as well as provide an efficient alternative to hiring voice actors to dub individual films or television shows.
Another company, the UK-based Synthesia, is using AI-based video-editing software to improve the quality of dubbing by tweaking the actor’s mouth movements to fit the local language they are speaking.
Meanwhile, WellSaid Labs—a Seattle-based startup—is offering AI voices with different voice personalities (such as “energetic and insightful” or “poised and expressive”) to its clients.
What’s the technology behind these products?
While synthetic voices have been around for a while, recent breakthroughs in deep learning have made replicating the subtleties of human speech easier than ever.
Previously, synthetic voices were just words glued together to produce a clunky, robotic effect. Nowadays, voice developers no longer need to provide the exact pacing, pronunciation, or intonation of the generated speech themselves. Instead, they simply fill hours of audio into an algorithm that deciphers those patterns for itself. These voices can change in style and emotion, take pauses and breathe in the correct places, and learn the tone and patterns of the speech of an actor.
The voice engine constructed by WellSaid consists of two primary deep-learning models: the first predicts what a speaker will sound like from a passage of text, and the second fills in the details—including breaths and the way the voice would resonate in a particular environment.
Traditionally, dubbing is done without making any alterations to the source video—which means that local script writers have to struggle to make the dialogue match the action taking place on the screen, and voice actors have to deliver their dialogue perfectly to make sure the dub doesn’t feel off. The time to write these scripts and to dub the movie is significantly shorter when artificial intelligence is used to simply change the lip movement of the actor post production (which also makes the scene more realistic to a viewer). While Synthesia’s deep learning algorithm has perfected making it look like an actor’s mouth is forming foreign words, it still isn’t capable of manufacturing the facial idiosyncrasies of individual actors—which would likely change depending on the language the actor is speaking in.
Why does this innovation matter?
AI voices are increasing in popularity amongst brands who want to maintain a consistent sound in millions of interactions with customers. Its popularity has also increased due to the omnipresence of smart speakers, automated service, and digital assistants today—that require brands to produce more than a hundred hours of audio every month.
It’s also getting increasingly vital for film and television producers to take their products to international markets. China’s influence on the global box office has grown exponentially in recent years; in 2020, a vast majority of Netflix’s subscriber gains were from outside the United States; and one-third of Disney+ subscribers are in India.
Traditional dubbing is an expensive and time-consuming process, and AI voices provide a cheap and scalable alternative to them. With DeepDub, an eight-episode television series—that would ordinarily take 14-16 weeks to dub—can be dubbed in just six weeks. This quicker turnaround for dubbed content could also help reduce piracy—which often occurs because of the time window between content released in the United States and other regions. The bigger the time window is, the greater the threat of piracy.
