'You Talking to Me?' This New Startup Uses Deepfake Technology for Movie Dubbing

Sam Blake

Sam primarily covers entertainment and media for dot.LA. Previously he was Marjorie Deane Fellow at The Economist, where he wrote for the business and finance sections of the print edition. He has also worked at the XPRIZE Foundation, U.S. Government Accountability Office, KCRW, and MLB Advanced Media (now Disney Streaming Services). He holds an MBA from UCLA Anderson, an MPP from UCLA Luskin and a BA in History from University of Michigan. Email him at samblake@dot.LA and find him on Twitter @hisamblake

'You Talking to Me?' This New Startup Uses Deepfake Technology for Movie Dubbing

Robert De Niro's performance seemed totally off. In fact, nothing about the movie was working.

The problem, as Scott Mann saw it, was that the dubbing process had created a dissonance between the original film and the translated version he was watching, the cumulative effect of which was that it had "become a different film." That was Mann's opinion when he first saw a translated version of "Heist," the 2015 film about a casino robbery that he directed.

"I saw the foreign translations of my film, and I was horrified," said Mann, who lives in the Hollywood Hills. "As filmmakers we don't tend to watch our own movies in foreign languages and when I was made aware of it, it really shocked me."

Film director Scott Mann (left) and Robert De Niro on the set of "Heist".Film director Scott Mann (left) and Robert De Niro on the set of "Heist".

International markets have become more important than ever for film and television. Projects rely on foreign audiences to recoup their costs; as the U.S. market grows saturated, streaming services are increasingly competing for overseas viewers.

Netflix, for instance, consistently sees foreign-produced projects like "Money Heist" (aka "La Casa de Papel") and "Lupin" as among its most popular shows. Last year "Parasite" became the first foreign-language film to win an Oscar for Best Picture.

As Mann sought a better solution for his films, he says he came across a research paper that described a new artificial intelligence solution to the dubbing problem. On Monday, he launched a company that uses the technology, and which he hopes will become the new standard in Hollywood and beyond.

Flawless, which Mann co-founded with Nick Lynes of his native England, uses technology that was developed by a team of researchers at Germany's Max Planck Institute. In the paper detailing their "style-preserving" approach, the group explains how their technique improves upon the dubbing technology the industry relies on.

Traditional dubbing techniques can be categorized into two groups, they write. In the audio-only approach, a foreign-language "source" actor recites the translated script and the resulting audio file is merged with the original video of the "target" actor. The translated script must be rewritten, of course, and is often done so to try to preserve alignment between the lip movements of the target actor and the audio footage from the source's translation. For example, when the target actor pronounces a close-mouthed "b", "m" or "p" sound in the original, the translated script aims to use a foreign word that requires a similar lip movement. But working within these constraints can cause the script's nuances to get literally lost in translation.

The second approach, called "visual dubbing," changes the visual footage of the original performance, so that the actor's lip movements align with the translated audio. The downside, write the researchers, is that doing so "removes the person-specific idiosyncrasies and the style of the target actor, and makes the target actor's face move unnaturally like the source actor."

The new technology uses artificial intelligence to transfer the source actor's lip movements so that, by contrast, it matches the target actor's characteristic style. In doing so, the company hopes it will appear to viewers that De Niro himself is actually speaking the dubbed foreign language.

How Flawless Works

The startup uses technology similar to "deepfake" techniques that manipulate video footage to make a subject appear to be saying something he or she is not. In Flawless' case the technology is used in service of a more realistic translation of the film, rather than, say, creating an embarrassing moment for a politician or celebrity.

To achieve that, its artificial intelligence and machine learning model analyzes video footage of the actor – De Niro, in this example – to learn the subtle nuances of how his lip movements relate to other features of his face, including head position and eye motion. On the other end, it analyzes the translator's performance, then generates new footage that maintains De Niro's style while incorporating the lip movements of the translator. The new footage is rendered exclusively on the actor's (De Niro's) face, and keeps the original background unchanged.

The original researchers write that five-minute videos are sufficient for training their "style translator." Mann added that "for 'locked/completed' productions, the process takes around 10 weeks from ingestion to finals."

The company has signed its first contract but is "sworn to secrecy on the details," Mann said, adding that pricing will be based on the number of translations and the length of the film.

The company has so far been self-financed. It received some external funding in 2020, but won't disclose how much. Lynes wrote it will be raising a Series A round of funding later this year.

In their paper, the researchers cautioned that their technology could be used in pernicious ways as well. They say they have been working on AI technology "to detect synthetically generated or edited video at high precision to make it easier to spot forgeries."

Despite that potential, Mann thinks the technology could "completely change filmmaking" – well beyond dubbing.

"What you're constantly doing as a filmmaker is you're doing the same thing again and again from different angles," he said."From a production point of view, that's very time consuming and expensive. What this allows, is you could get one great take, from one angle and then essentially have it recreated from other angles."

He also thinks it could make remaking foreign films a thing of the past.

"Rather than having to remake a movie in English and all these different languages, we'll be able to enjoy the original," he said. "I think that will have an effect on the international community of filmmaking, and all types of actors that really should be on the world stage but currently aren't because no one speaks that language."

"I do think this is the tip of the iceberg on this technology in the film field," he added.

https://twitter.com/hisamblake
samblake@dot.la

Subscribe to our newsletter to catch every headline.

🏰 Disney's Epic Investment Stands Out Amidst Gaming Industry Layoffs

Christian Hetrick

Christian Hetrick is dot.LA's Entertainment Tech Reporter. He was formerly a business reporter for the Philadelphia Inquirer and reported on New Jersey politics for the Observer and the Press of Atlantic City.

🔦 Spotlight

In the midst of widespread gaming industry layoffs, a glimmer of positive news emerges as Disney announces a significant move: a $1.5 billion investment in Epic Games. 🏰💰🐭

Image Source: Disney

Disney's $1.5 billion investment in Epic Games, disclosed late Wednesday, signals a strategic alignment aimed at expanding the success of "Fortnite." The deal enhances Epic's growth prospects after financial setbacks, including layoffs, and strengthens the partnership between the two companies. With Disney gaining a larger equity stake in Epic, the collaboration will broaden the integration of beloved Disney franchises like Marvel, Star Wars, Pixar, and Avatar into the game, potentially boosting its appeal and longevity. This significant investment underscores Disney's commitment to interactive entertainment and signifies a shift towards games as a primary revenue stream, aligning with the growing trend of digital engagement among younger demographics. Moreover, the potential for crossover sales of physical Disney products within "Fortnite" and the exploration of new content distribution channels are just some of the opportunities arising from this partnership.

For LA tech, the Disney-Epic Games partnership represents a validation of the region's burgeoning tech and gaming ecosystem. The substantial investment in Epic, who maintains a large Los Angeles office with 1,000+ employees (according to LinkedIn), reflects confidence in the LA’s talent pool and innovation potential. Additionally, this partnership between two industry giants fosters an environment for further collaboration, investment, and growth within LA's tech sector. As Disney and Epic Games deepen their ties and explore new avenues for content integration and distribution, it not only elevates the prominence of LA as a tech hub but also stimulates economic growth and job creation in the region. This partnership highlights LA's unique position as a hub where technology and entertainment converge. With its ability to integrate diverse industries, LA is driving innovation and expansion in digital entertainment. 🚀💸🎮

🤝 Venture Deals

LA Companies

  • ProducePay, a financing and marketplace platform for the fresh produce market, raised a $38M Series D led by Syngenta Group Ventures joined by Commonfund, Highgate Private Equity, G2 Venture Partners, Anterra Capital, Astanor Ventures, Endeavor8, Avenue Venture Opportunities, Avenue Sustainable Solutions, and Red Bear Angels. - learn more
  • Blush, an invite-only dating app that drives users to local businesses on dates, raised a $7M Seed Round from individuals like Naval Ravikant. - learn more
  • Mogul, a startup founded last year that provides an overview of an artist's royalty earnings and identifies areas where money is owed but has not yet been collected, raised a $1.9 million seed round from Wonder Ventures, United Talent Agency, AmplifyLA, and Creator Partners. - learn more
  • Avnos, a hybrid direct air capture startup, raised a $36M Series A led by NextEra Energy and joined by Safran Corporate Ventures, Shell Ventures, Envisioning Partners, and Rusheen Capital Management. - learn more
  • AI.fashion, startup whose mission is to help retailers enhance the online shopping experience by providing consumers with virtual try-ons and personalized fashion recommendations, raised a $3.6M Seed Round led by Neo. - learn more
  • Suma Wealth, startup that aims to demystify financial topics and provide culturally relevant content, virtual experiences, and resources to help Latino users navigate financial challenges and opportunities, raised a $2.2M Seed Round . Radicle Impact led, and was joined by Vamos Ventures, OVO fund and the American Heart Association Impact Fund. - learn more
  • 222, a startup that helps users discover their city and meet new people through unique social experiences, raised a $2.5M Seed Round. Investors included 1517 Fund, General Catalyst, Best Nights VC, Scrum Ventures, and Upfront Ventures. - learn more
  • LimaCharlie, a security operations cloud platform, raised a $10.2M Series A led by Sands Capital. - learn more
  • Polycam, an app that uses a smartphone’s sensors to capture 3D scans of objects, raised an $18M Series A co-led by Left Lane Capital and Adjacent, and joined by Adobe Ventures and individuals like Chad Hurley and Shaun Maguire. -learn more.

LA Venture Funds

Actively Raising

  • ReelCall, Inc., an entertainment technology company focused on powerful apps and platforms that help build and maintain the professional network of connections vital to career growth, is raising a $850K Pre-Seed Round. - learn more
  • CZero, a startup building software to decarbonize logistics for logistics businesses and goods business through a vetted marketplace and optimization software. - learn more
  • Couri, a technology startup addressing last-mile delivery issues, is raising a $450K Pre-Seed Round at a $2.2M post money valuation. - learn more
  • Sweetie, a marketplace to help people plan date nights, is raising a $1.5M Pre Seed Round. - learn more
  • StartupStarter, an investment platform that provides real-time data and analytics on startups, is raising an $850K Angel Round. - learn more

If you’re a founder raising money in Los Angeles, give us a shout, and we’d love to include you in the newsletter!

Venture Waves, Climate Tech Wins, and Silicon Beach's Ongoing Evolution

Christian Hetrick

Christian Hetrick is dot.LA's Entertainment Tech Reporter. He was formerly a business reporter for the Philadelphia Inquirer and reported on New Jersey politics for the Observer and the Press of Atlantic City.

Anduril Seeks $1.5B in VC Funds

Defense company Anduril Industries Inc., based in Costa Mesa and founded by Palmer Luckey, is seeking to raise $1.5 billion in fresh funds to boost its valuation to $12.5 billion or more, according to sources quoted by The Information. This fundraising effort, if successful, would mark one of the largest venture capital rounds of the year.

Image Source: Anduril

Anduril recently secured a contract to develop and test small unmanned fighter jet prototypes under the Air Force’s Collaborative Combat Aircraft (CCA) program, beating out major defense companies like Boeing, Lockheed Martin, and Northrop Grumman. Alongside General Atomics, Anduril will design, manufacture, and test these aircraft, with a final multibillion-dollar production decision expected in fiscal year 2026. This program aims to deliver at least 1,000 combat aircraft to fly in concert with manned platforms and is part of the Air Force’s Next Generation Air Dominance initiative. Central to Anduril’s success in this contract is the Fury autonomous air vehicle, acquired through the purchase of Blue Force Technologies. This victory underscores Anduril's rapid advancement in the defense sector, aligning with Luckey's vision of building faster and more cost-effective defense assets. - learn more

Los Angeles Ranks Number 1 in Emerging Climate Tech Hub

The 2024 Emerging Climate Tech Hubs Report by Revolution highlights Los Angeles as a burgeoning center for climate tech innovation. LA's growth in this sector is driven by its diverse talent pool, strong research institutions, and a culture of environmental consciousness. The city's unique mix of legacy industries, such as entertainment and aerospace, alongside emerging tech companies, positions it as a pivotal player in the climate tech landscape. This shift reflects a broader trend of decentralized climate tech funding across the U.S., reducing the historical dominance of California's traditional hubs. - learn more

Silicon Beach: Looking Back, Moving Forward

Assessing the overall health of the startup market is challenging, especially as venture capital funding has decreased by an average of 61% from 2021 to 2023 across the top VC markets in the US. Markets with robust ecosystems in AI, SaaS, Biotech, Healthtech, and Fintech appear to be weathering the downturn better than those focused on Consumer and Gaming industries, areas where Los Angeles traditionally excels.

Percent Change In VC Funding By Region

CB Insights

LA Times paints a rather bleak outlook on the Los Angeles tech scene noting venture capital funding in Greater Los Angeles plummeted 73% from 2021 to 2022. Silicon Beach, once a vibrant tech corridor, currently faces high vacancy rates and lacks late-stage financiers, especially in the AI sector. However, there are positive signs, including growth in aerospace startups and increased venture capital investment in early 2024, suggesting a potential rebound for LA's tech ecosystem.

While LA may not be exceeding expectations during this period, its tech ecosystem warrants a nuanced evaluation, given the broader market dynamics and its strong performance in specific sectors. Reach out to us with your thoughts.

🚀 SpaceX gears up for another stellar year, active raises, and more

Christian Hetrick

Christian Hetrick is dot.LA's Entertainment Tech Reporter. He was formerly a business reporter for the Philadelphia Inquirer and reported on New Jersey politics for the Observer and the Press of Atlantic City.

Happy Friday Los Angeles! You made it through the first week of 2024!

🔦 Spotlight

Elon Musk may be a divisive (albeit entertaining) figure, but the continued success of SpaceX is pivotal for the aerospace industry in Los Angeles and more broadly around the world.

Image Source: SpaceX webcast

What happened with SpaceX in 2023?

  • Elon Musk challenged Facebook founder, Mark Zuckerberg to a cage fight.
  • SpaceX launched 96 successful missions with its Falcon series of rockets, a 57% increase over its previous annual record.
  • SpaceX conducted two test flights of the largest and most powerful rocket ever built, Starship.
  • Roughly two-thirds of SpaceX's launches in 2023 were devoted to building out Starlink, the company's satellite-internet megaconstellation.
  • Isaacson’s Elon Musk biography was published in September including everything from Musk’s tumultuous relationship with his father to his work ethic and “demon mode”.

Moving forward what can we expect from SpaceX and its controversial founder? Continued innovation pushing the aerospace industry to new limits? Yes. More drama? Without a doubt.

Here is some of what is to come in 2024:

🤝 Venture Deals

Just Announced

Check back next week!

LA Exits

  • CG Oncology, an Irvine, CA-based developer of immunotherapies for bladder cancer, filed for a $100M IPO. It plans to list on the Nasdaq (CGON) with Morgan Stanley as left lead underwriter, and has raised around $317m in VC funding. - learn more
  • McNally Capital agreed to sell Advanced Micro Instruments, a Costa Mesa, CA-based maker of gas analyzers and sensing technologies, to Enpro (NYSE: NPO). - learn more

Actively Raising

  • ReelCall, Inc., an entertainment technology company focused on powerful apps and platforms that help build and maintain the professional network of connections vital to career growth, is raising a $850K Pre-Seed Round. - learn more
  • CZero, a hard-tech startup that is developing a technology for decarbonizing natural gas, is raising a $1.5M Seed Round. - learn more
  • Couri, a technology startup addressing last-mile delivery issues, is raising a $450K Pre-Seed Round at a $2.2M post money valuation. - learn more
  • Sweetie, a marketplace to help people plan date nights, is raising a $250K Angel Round. - learn more
  • StartupStarter, an investment platform that provides real-time data and analytics on startups, is raising an $850K Angel Round. - learn more

If you’re a founder raising money in Los Angeles, give us a shout, and we’d love to include you in the newsletter!

📅 LA Tech Calendar

Sunday, January 7th

Wednesday, January 10th

  • Startup Cafe: Networking with a Kick - Entrepreneurs, Startups, and Tech Enthusiasts join together to meet and connect with like-minded people, industry professionals and investors, while enjoying a nice cup of coffee in Venice at The KINN. This week’s interactive discussion about AI’s evolution in entertainment will feature Dr. Sam Khoze and Rachel Joy Victor.
  • Venice Tech Happy Hour- Join Startup Coil and FoundrHaus Wednesday evening and enjoy the sunset from the rooftop, grab a bite overlooking Abbot Kinney, and mingle with other tech enthusiasts and entrepreneurs by the bar on the patio.

Have an awesome event coming up? Reach out to be featured on next week’s Newsletter!

📙 What We’re Reading

RELATEDEDITOR'S PICKS
Trending