Robert De Niro's performance seemed totally off. In fact, nothing about the movie was working.
The problem, as Scott Mann saw it, was that the dubbing process had created a dissonance between the original film and the translated version he was watching, the cumulative effect of which was that it had "become a different film." That was Mann's opinion when he first saw a translated version of "Heist," the 2015 film about a casino robbery that he directed.
"I saw the foreign translations of my film, and I was horrified," said Mann, who lives in the Hollywood Hills. "As filmmakers we don't tend to watch our own movies in foreign languages and when I was made aware of it, it really shocked me."
Film director Scott Mann (left) and Robert De Niro on the set of "Heist".
International markets have become more important than ever for film and television. Projects rely on foreign audiences to recoup their costs; as the U.S. market grows saturated, streaming services are increasingly competing for overseas viewers.
Netflix, for instance, consistently sees foreign-produced projects like "Money Heist" (aka "La Casa de Papel") and "Lupin" as among its most popular shows. Last year "Parasite" became the first foreign-language film to win an Oscar for Best Picture.
As Mann sought a better solution for his films, he says he came across a research paper that described a new artificial intelligence solution to the dubbing problem. On Monday, he launched a company that uses the technology, and which he hopes will become the new standard in Hollywood and beyond.
Flawless, which Mann co-founded with Nick Lynes of his native England, uses technology that was developed by a team of researchers at Germany's Max Planck Institute. In the paper detailing their "style-preserving" approach, the group explains how their technique improves upon the dubbing technology the industry relies on.
Traditional dubbing techniques can be categorized into two groups, they write. In the audio-only approach, a foreign-language "source" actor recites the translated script and the resulting audio file is merged with the original video of the "target" actor. The translated script must be rewritten, of course, and is often done so to try to preserve alignment between the lip movements of the target actor and the audio footage from the source's translation. For example, when the target actor pronounces a close-mouthed "b", "m" or "p" sound in the original, the translated script aims to use a foreign word that requires a similar lip movement. But working within these constraints can cause the script's nuances to get literally lost in translation.
The second approach, called "visual dubbing," changes the visual footage of the original performance, so that the actor's lip movements align with the translated audio. The downside, write the researchers, is that doing so "removes the person-specific idiosyncrasies and the style of the target actor, and makes the target actor's face move unnaturally like the source actor."
The new technology uses artificial intelligence to transfer the source actor's lip movements so that, by contrast, it matches the target actor's characteristic style. In doing so, the company hopes it will appear to viewers that De Niro himself is actually speaking the dubbed foreign language.
How Flawless Works
The startup uses technology similar to "deepfake" techniques that manipulate video footage to make a subject appear to be saying something he or she is not. In Flawless' case the technology is used in service of a more realistic translation of the film, rather than, say, creating an embarrassing moment for a politician or celebrity.
To achieve that, its artificial intelligence and machine learning model analyzes video footage of the actor – De Niro, in this example – to learn the subtle nuances of how his lip movements relate to other features of his face, including head position and eye motion. On the other end, it analyzes the translator's performance, then generates new footage that maintains De Niro's style while incorporating the lip movements of the translator. The new footage is rendered exclusively on the actor's (De Niro's) face, and keeps the original background unchanged.
The original researchers write that five-minute videos are sufficient for training their "style translator." Mann added that "for 'locked/completed' productions, the process takes around 10 weeks from ingestion to finals."
The company has signed its first contract but is "sworn to secrecy on the details," Mann said, adding that pricing will be based on the number of translations and the length of the film.
The company has so far been self-financed. It received some external funding in 2020, but won't disclose how much. Lynes wrote it will be raising a Series A round of funding later this year.
In their paper, the researchers cautioned that their technology could be used in pernicious ways as well. They say they have been working on AI technology "to detect synthetically generated or edited video at high precision to make it easier to spot forgeries."
Despite that potential, Mann thinks the technology could "completely change filmmaking" – well beyond dubbing.
"What you're constantly doing as a filmmaker is you're doing the same thing again and again from different angles," he said."From a production point of view, that's very time consuming and expensive. What this allows, is you could get one great take, from one angle and then essentially have it recreated from other angles."
He also thinks it could make remaking foreign films a thing of the past.
"Rather than having to remake a movie in English and all these different languages, we'll be able to enjoy the original," he said. "I think that will have an effect on the international community of filmmaking, and all types of actors that really should be on the world stage but currently aren't because no one speaks that language."
"I do think this is the tip of the iceberg on this technology in the film field," he added.
Alexa-maker Amazon is creating a machine learning and artificial intelligence research lab at USC as the retail giant grapples with growing privacy concerns around its products. The Center for Secure and Trusted Machine Learning, part of USC's Viterbi School of Engineering, will support research that looks at new ways to secure and preserve privacy in machine learning and can be applied at scale "to support billions of users."
Amazon's artificial intelligence systems extend beyond its smart home devices; the company automates much of its processes using machine learning; including product recommendation, the Amazon Echo and the Amazon Go store (a brick and mortar location that runs without cashiers). Amazon also recently launched Halo, a wearable fitness tracker comparable to the Fitbit that also connects with Alexa and the rest of its smart devices.
A.I. and machine learning underpin almost all Amazon's products, and the technology is what powers any smart home device. The "internet of things" concept -- the idea that different individual computers can communicate with one another on a universal network -- is also powered by A.I. Nearly all big tech companies use A.I.
"A.I. fuels just about everything we do at Amazon, and we challenge ourselves every day to find ways to use this technology to benefit customers," a company spokesperson told dot.LA. "A.I. is a key part of our culture because we are customer obsessed, and these technologies have developed as great tools for developing and improving customer experiences."
Amazon wouldn't comment on if it will use this research to develop its A.I.-enabled products, like the Alexa smart device.
Though it has relationships with other colleges, the program with USC is Amazon's first machine learning-focused fellowship project with a campus.
The center's goal is to make A.I. and machine learning technologies more secure and trusted by the public. It will be directed by Salman Avestimehr, professor of computer and electrical engineering at USC, who will also oversee related fellowships and the overall project.
Avestimehr said he thinks there are many companies besides Amazon that could benefit from the center's research.
"Amazon is interested in this, and many others. [Machine learning] is a hot topic, and it's on everybody's mind," Avestimehr said. "Privacy, security and trust resonates with everybody."
Salman Avestimehr is a professor of computer and electrical engineering at USC.
Privacy, according to Avestimehr, refers to keeping individual users' data safe, while security is related to securing the open-source systems from threats. "Since everybody can be a part of this ecosystem of machine learning, therefore it is also open to any adversary behavior," he added.
There's also the challenge big tech companies face in getting their customers to fully trust their automated systems (and keep using their devices).
Google is another tech giant that's trying to figure out how to approach and market A.I., which it uses in many facets of its business including its Google Home devices, which compete with Amazon's Alexa. Lead researchers and engineers at Google have quit over concerns the company and its CEO Sundar Pichai aren't prioritizing diversity in developing A.I. -- an issue they've voiced since 2015. Google's co-head of ethical A.I. Margaret Mitchell is currently under investigation for allegedly sharing classified Google documents with outside sources.
"If this is something like coming up with this algorithm to run your home, how would you trust that? How do you trust this algorithm that is learning by itself?," Avestimehr said.
Amazon had similar issues. Cybersecurity researchers including those at Check Point have uncovered privacy concerns with the Alexa, including the ability to hack into the device, steal personal information and change which "skills" Alexa can perform. "Successful exploitation would have required just one click," Check Point wrote in its report.
At CES last year, Amazon said it sold at least 200 million Alexa devices to date, and that its customers use the voice assistant prompts to control their smart homes a combined "hundreds of millions of times" each week.
"At Amazon, privacy and security are foundational," an Amazon spokesperson said. "Our highest priorities are keeping customers' information safe, providing customers with transparency and control, and making privacy controls incredibly easy to use and understand."
The technology and research produced by the lab could lead to a wider understanding of how A.I. and machine learning works. Avestimehr said he hopes it'll also convince the public to engage with more complex A.I. systems that could actually be dangerous, like autonomous vehicles.
"They're not making big decisions yet," Avestimehr said of most current A.I. systems.
Under Avestimehr's direction, the center will accept qualified USC PhD candidates into its Amazon Machine Learning Fellows program, where they will gain access to funded research projects, annual fellowships, public research symposiums and annual workshops. The program will also reach out to younger engineers; there are plans to train and eventually recruit high school and university students.
"Related to our university, it's good at attracting talent, educating talent and these fellowship resources will be very useful to drawing talented students, educating them and [also] recognizing the greatest students we have at USC," Avestimehr said.
Amazon and USC have partnered before on projects. The ecommerce giant said through a spokesperson that it chose to work with USC to develop a machine learning center partly because it deepens Amazon's access to the graduate talent pool.
"We are delighted to bring together top talent at Amazon and USC in a joint mission to drive ground-breaking advances in privacy and security preserving machine learning; advances that enable us to continue to safely and securely deliver experiences," Amazon's Alexa AI Vice President — and former USC vice dean of engineering — Prem Natarajan said in a statement.
Though it's headquartered in Seattle, Amazon has a sizable operation in Los Angeles. The company said it continues to hire at its hub in L.A., and added, "there are currently more than 500 tech and corporate roles available." Right now, Amazon said it employs thousands in L.A. County.