'You Talking to Me?' This New Startup Uses Deepfake Technology for Movie Dubbing

Sam Blake

Sam primarily covers entertainment and media for dot.LA. Previously he was Marjorie Deane Fellow at The Economist, where he wrote for the business and finance sections of the print edition. He has also worked at the XPRIZE Foundation, U.S. Government Accountability Office, KCRW, and MLB Advanced Media (now Disney Streaming Services). He holds an MBA from UCLA Anderson, an MPP from UCLA Luskin and a BA in History from University of Michigan. Email him at samblake@dot.LA and find him on Twitter @hisamblake

'You Talking to Me?' This New Startup Uses Deepfake Technology for Movie Dubbing

Robert De Niro's performance seemed totally off. In fact, nothing about the movie was working.

The problem, as Scott Mann saw it, was that the dubbing process had created a dissonance between the original film and the translated version he was watching, the cumulative effect of which was that it had "become a different film." That was Mann's opinion when he first saw a translated version of "Heist," the 2015 film about a casino robbery that he directed.

"I saw the foreign translations of my film, and I was horrified," said Mann, who lives in the Hollywood Hills. "As filmmakers we don't tend to watch our own movies in foreign languages and when I was made aware of it, it really shocked me."

Film director Scott Mann (left) and Robert De Niro on the set of "Heist".Film director Scott Mann (left) and Robert De Niro on the set of "Heist".

International markets have become more important than ever for film and television. Projects rely on foreign audiences to recoup their costs; as the U.S. market grows saturated, streaming services are increasingly competing for overseas viewers.

Netflix, for instance, consistently sees foreign-produced projects like "Money Heist" (aka "La Casa de Papel") and "Lupin" as among its most popular shows. Last year "Parasite" became the first foreign-language film to win an Oscar for Best Picture.

As Mann sought a better solution for his films, he says he came across a research paper that described a new artificial intelligence solution to the dubbing problem. On Monday, he launched a company that uses the technology, and which he hopes will become the new standard in Hollywood and beyond.

Flawless, which Mann co-founded with Nick Lynes of his native England, uses technology that was developed by a team of researchers at Germany's Max Planck Institute. In the paper detailing their "style-preserving" approach, the group explains how their technique improves upon the dubbing technology the industry relies on.

Traditional dubbing techniques can be categorized into two groups, they write. In the audio-only approach, a foreign-language "source" actor recites the translated script and the resulting audio file is merged with the original video of the "target" actor. The translated script must be rewritten, of course, and is often done so to try to preserve alignment between the lip movements of the target actor and the audio footage from the source's translation. For example, when the target actor pronounces a close-mouthed "b", "m" or "p" sound in the original, the translated script aims to use a foreign word that requires a similar lip movement. But working within these constraints can cause the script's nuances to get literally lost in translation.

The second approach, called "visual dubbing," changes the visual footage of the original performance, so that the actor's lip movements align with the translated audio. The downside, write the researchers, is that doing so "removes the person-specific idiosyncrasies and the style of the target actor, and makes the target actor's face move unnaturally like the source actor."

The new technology uses artificial intelligence to transfer the source actor's lip movements so that, by contrast, it matches the target actor's characteristic style. In doing so, the company hopes it will appear to viewers that De Niro himself is actually speaking the dubbed foreign language.

How Flawless Works

The startup uses technology similar to "deepfake" techniques that manipulate video footage to make a subject appear to be saying something he or she is not. In Flawless' case the technology is used in service of a more realistic translation of the film, rather than, say, creating an embarrassing moment for a politician or celebrity.

To achieve that, its artificial intelligence and machine learning model analyzes video footage of the actor – De Niro, in this example – to learn the subtle nuances of how his lip movements relate to other features of his face, including head position and eye motion. On the other end, it analyzes the translator's performance, then generates new footage that maintains De Niro's style while incorporating the lip movements of the translator. The new footage is rendered exclusively on the actor's (De Niro's) face, and keeps the original background unchanged.

The original researchers write that five-minute videos are sufficient for training their "style translator." Mann added that "for 'locked/completed' productions, the process takes around 10 weeks from ingestion to finals."

The company has signed its first contract but is "sworn to secrecy on the details," Mann said, adding that pricing will be based on the number of translations and the length of the film.

The company has so far been self-financed. It received some external funding in 2020, but won't disclose how much. Lynes wrote it will be raising a Series A round of funding later this year.

In their paper, the researchers cautioned that their technology could be used in pernicious ways as well. They say they have been working on AI technology "to detect synthetically generated or edited video at high precision to make it easier to spot forgeries."

Despite that potential, Mann thinks the technology could "completely change filmmaking" – well beyond dubbing.

"What you're constantly doing as a filmmaker is you're doing the same thing again and again from different angles," he said."From a production point of view, that's very time consuming and expensive. What this allows, is you could get one great take, from one angle and then essentially have it recreated from other angles."

He also thinks it could make remaking foreign films a thing of the past.

"Rather than having to remake a movie in English and all these different languages, we'll be able to enjoy the original," he said. "I think that will have an effect on the international community of filmmaking, and all types of actors that really should be on the world stage but currently aren't because no one speaks that language."

"I do think this is the tip of the iceberg on this technology in the film field," he added.

https://twitter.com/hisamblake
samblake@dot.la

Subscribe to our newsletter to catch every headline.

How the 'Thrift Haul' Trend Boosted Secondhand Ecommerce Platforms

Lon Harris
Lon Harris is a contributor to dot.LA. His work has also appeared on ScreenJunkies, RottenTomatoes and Inside Streaming.
How the 'Thrift Haul' Trend Boosted Secondhand Ecommerce Platforms
Evan Xie

If you can believe it, it’s been more than a decade since rapper Macklemore extolled the virtues of thrift shopping in a viral music video. But while scouring the ranks of vintage clothing stores looking for the ultimate come-up may have waned in popularity since 2012, the online version of this activity is apparently thriving.

According to a new trend story from CNBC, interest in “reselling” platforms like Etsy-owned Depop and Poshmark has exploded in the years since the start of the COVID-19 pandemic and lockdown. In an article that spends a frankly surprising amount of time focused on sellers receiving death threats before concluding that they’re “not the norm,” the network cites the usual belt-tightening ecommerce suspects – housebound individuals doing more of their shopping online coupled with inflation woes and recession fears – as the causes behind the uptick.

As for data, there’s a survey from Depop themselves, finding that 53% of respondents in the UK are more inclined to shop secondhand as living costs continue to rise. Additional research from Advance Market Analytics confirms the trend, citing not just increased demand for cheap clothes but the pressing need for a sustainable alternative to recycling clothing materials at its core.

The major popularity of “thrift haul” videos across social media platforms like YouTube and TikTok has also boosted the visibility of vintage clothes shopping and hunting for buried treasures. Teenage TikToker Jacklyn Wells scores millions of views on her thrift haul videos, only to get routinely mass-accused of greed for ratching up the Depop resell prices for her coolest finds and discoveries. Nonetheless, viral clips like Wells’ have helped to embed secondhand shopping apps more generally within online fashion culture. Fashion and beauty magazine Hunger now features a regular list of the hottest items on the re-sale market, with a focus on how to use them to recreate hot runway looks.

As with a lot of consumer and technology trends, the sudden surge of interest in second-hand clothing retailers was only partly organic. According to The Drum, ecommerce apps Vinted, eBay, and Depop have collectively spent around $120 million on advertising throughout the last few years, promoting the recent vintage shopping boom and helping to normalize second-hand shopping. This includes conventional advertising, of course, but also deals with online influencers to post content like “thrift haul” videos, along with shoutouts for where to track down the best finds.

Reselling platforms have naturally responded to the increase in visibility with new features (as well as a predictable hike in transaction fees). Poshmark recently introduced livestreamed “Posh Shows” during which sellers can host auctions or provide deeper insight into their inventory. Depop, meanwhile, has introduced a “Make Offer” option to fully integrate the bartering and negotiation process into the app, rather than forcing buyers and sellers to text or Direct Message one another elsewhere. (The platform formerly had a comments section on product pages, but shut this option down after finding that it led to arguments, and wasn’t particularly helpful in making purchase decisions.)

Now that it’s clear there’s money to be made in online thrift stores, larger and more established brands and retailers are also pushing their way into the space. H&M and Target have both partnered with online thrift store ThredUp on featured collections of previously-worn clothing. A new “curated” resale collection from Tommy Hilfiger – featuring minorly damaged items that were returned to its retail stores – was developed and promoted through a partnership with Depop, which has also teamed with Kellogg’s on a line of Pop-Tarts-inspired wear. J.Crew is even bringing back its classic ‘80s Rollneck Sweater in a nod to the renewed interest in all things vintage.

Still, with any surge of popularity and visibility, there must also come an accompanying backlash. In a sharp editorial this week for Arizona University’s Daily Wildcat, thrift shopping enthusiast Luke Lawson makes the case that sites like Depop are “gentrifying fashion,” stripping communities of local thrift stores that provide a valuable public service, particularly for members of low-income communities. As well, UK tabloids are routinely filled with secondhand shopping horror stories these days, another evidence point as to their increased visibility among British consumers specifically, not to mention the general dangers of buying personal items from strangers you met over the internet.

How to Startup: Mission Acquisition

Spencer Rascoff

Spencer Rascoff serves as executive chairman of dot.LA. He is an entrepreneur and company leader who co-founded Zillow, Hotwire, dot.LA, Pacaso and Supernova, and who served as Zillow's CEO for a decade. During Spencer's time as CEO, Zillow won dozens of "best places to work" awards as it grew to over 4,500 employees, $3 billion in revenue, and $10 billion in market capitalization. Prior to Zillow, Spencer co-founded and was VP Corporate Development of Hotwire, which was sold to Expedia for $685 million in 2003. Through his startup studio and venture capital firm, 75 & Sunny, Spencer is an active angel investor in over 100 companies and is incubating several more.

How to Startup: Mission Acquisition

Numbers don’t lie, but often they don’t tell the whole story. If you look at the facts and figures alone, launching a startup seems like a daunting enterprise. It seems like a miracle anyone makes it out the other side.

  • 90% of startups around the world fail.
  • On average, it takes startups 2-3 years to turn a profit. (Venture funded startups take far longer.)
  • Post-seed round, fewer than 10% of startups go on to successfully raise a Series A investment.
  • Less than 1% of startups go public.
  • A startup only has a .00006% chance of becoming a unicorn.

Ouch.

Read moreShow less
https://twitter.com/spencerrascoff
https://www.linkedin.com/in/spencerrascoff/
admin@dot.la

From The Vault: VC Legend Bill Gurley On Startups, Venture Capital and Scaling

Spencer Rascoff

Spencer Rascoff serves as executive chairman of dot.LA. He is an entrepreneur and company leader who co-founded Zillow, Hotwire, dot.LA, Pacaso and Supernova, and who served as Zillow's CEO for a decade. During Spencer's time as CEO, Zillow won dozens of "best places to work" awards as it grew to over 4,500 employees, $3 billion in revenue, and $10 billion in market capitalization. Prior to Zillow, Spencer co-founded and was VP Corporate Development of Hotwire, which was sold to Expedia for $685 million in 2003. Through his startup studio and venture capital firm, 75 & Sunny, Spencer is an active angel investor in over 100 companies and is incubating several more.

Bill Gurley in a blue suit
Bill Gurley

This interview was originally published on December of 2020, and was recorded at the inaugural dot.LA Summit held October 27th & 28th.

One of my longtime favorite episodes of Office Hours was a few years ago when famed venture capitalist Bill Gurley and I talked about marketplace-based companies, how work-from-home will continue to accelerate business opportunities and his thoughts on big tech and antitrust.

Read moreShow less
https://twitter.com/spencerrascoff
https://www.linkedin.com/in/spencerrascoff/
admin@dot.la
RELATEDEDITOR'S PICKS
LA TECH JOBS
interchangeLA
Trending