'Avatar: The Way of Water' Innovative Tech Sails $2B Box Office, 4 Oscar Noms

This is the web version of dot.LA’s daily newsletter. Sign up to get the latest news on Southern California’s tech, startup and venture capital scene.

After years of skeptics questioning the original “Avatar’s” cultural footprint – and whether or not anyone could actually remember the names of any characters – filmmaker James Cameron seems to have at least temporarily silenced his many haters. His massively-budgeted long-awaited sci-fi sequel “Avatar: The Way of Water” crossed the $2 billion mark at the global box office this week, making it the sixth highest-grossing film in history. As if that weren’t achievement enough, the film has now been nominated for four Oscars, including Production Design, Sound, Visual Effects and Best Picture.

About That “Cultural Footprint”...

Competition in this year’s technical categories, including VFX, is particularly fierce. “Avatar 2” will face off for the Visual Effects prize against Netflix’s “All Quiet on the Western Front,” WB’s “The Batman,” and Disney-Marvel’s “Black Panther: Wakanda Forever,” but faces perhaps its strongest competition from 2022’s other mega-blockbuster: Paramount’s “Top Gun: Maverick.” (For all that film’s marketing around shooting practically in real fighter jets, it contains 2,400 VFX shots!)

While Academy voters take all kinds of factors into consideration when voting, Cameron’s film arguably represents the largest overall step forward in terms of new film technology, employing a variety of innovative solutions to bringing the distant and exotic planet of Pandora to fully-realized three-dimensional life. The original 2009 “Avatar” starred a cast of nine-foot-tall blue aliens running around a bioluminescent forest on an alien world, so it was already a pretty significant leap forward in terms of cinematic visuals. Nonetheless, the sequel posed an additional massive challenge that’s right there in the subtitle: “The Way of WATER.”

Under the (Virtual) Sea

Traditionally, big-budget VFX-heavy Hollywood films with underwater sequences shoot what’s called “dry for wet.” That is, the production remains on dry land, which is way easier on all the technology, and just use bluescreen backdrops along with wires to make it look like the actors and props are suspended beneath the waves. Cameron tried a few traditional wire-motion tests but ultimately concluded that it just wasn’t realistic enough for his movie.

Instead, Cameron worked with Australian cinematographer Pawel Achtel, who developed his own underwater camera rig known as DeepX 3D in 2015. This rig was then paired with underwater Nikon lenses from the 1980s, creating the only known camera system capable of capturing stereoscopic 3D while shooting underwater. Notably, the rig is also relatively light – weighing around 66 pounds – making it relatively maneuverable compared to conventional camera systems.

The actors filmed their underwater sequences in a 900,000-gallon tank, requiring them to master free diving and breath control. Action sequences were sometimes shot in a 250,000-gallon tank outfitted with a wave simulator that also captured coordinated shots both above and below the water line.

Putting It All Together

In a GQ profile – released as part of their “Men of the Year 2022” coverage – Cameron briefly discussed the overall process for assembling an “Avatar” film. First, he shoots what’s known as a “template,” a data-rich collection of visuals that captures the basic visual elements of a scene, including performances from human actors, lighting and camera movement. While shooting these sequences, the actors are wearing performance-capture suits covered in sensors to track their movement.

For “Avatar 2,” Cameron also employed additional facial cameras to capture tiny nuances of the actor’s expressions, giving the animators intricate details to translate to their alien Na’vi characters.

This template was then handed off to technicians, animators, and other creatives at Wētā FX – the New Zealand visual effects studio founded by Peter Jackson that designed his “Lord of the Rings” and “Hobbit” films – who added animated elements on top of the raw footage. This is significantly different from the kind of work effects artists would do on an entirely animated film of the kind released by DreamWorks or Pixar. As Cameron explained “the actors already defined what they did, but it has to be translated from the captured data to the 3D-CG character.”

Animation and Automation

Part of the animation process was automated using a complex network of algorithms, in large part to save time. While the original “Avatar” was—based on Cameron’s estimates— around 70-75% computer-generated, the new film is closer to 90% animation. One such technique uses flow maps to configure the layout of pores on characters’ faces. Others were required for realistically animating various water effects, making individual splashes and sprays look photorealistic and natural. Effects supervisor Eric Saindon estimates that there are 1,600 different water simulations visible in the finished film.

These kinds of digital shortcuts and tools are essential when you consider the sheer amount of work required to bring Pandora to fully-realied life on screen. According to Wētā artists, 57 new species of sea creature alone were created exclusively for the film. The company worked on 3,240 visual effects shots total, 2,225 of which included water.

Cameron’s VFX supervisors Joe Letteri and Richard Baneham also reimagined and upgraded the performance capture technology they’d developed for the original “Avatar,” known as FACS, or Facial Action Coding System. The new system utilizes a neural network that’s been trained on muscle movements in the human face, with specific attention to the ways that muscle, tissue, and skin move around one another.

Just Rig Things

Then, of course, there were the necessary camera upgrades. Cameron and his team famously designed a new “virtual” camera rig for the first “Avatar,” allowing him to visualize actors in the animated setting of Pandora while still on set. Never one to be outdone, Cameron developed an entirely new camera system for “Avatar 2” and its future sequels, incorporating multiple Sony Venice cameras into 3D stereoscopic rigs, capable of capturing images in vibrant 6K. When accompanied by cutting-edge lighting units and other new technologies, the full rig is capable of functionally simulating performance capture live on set, allowing the director to accurately visualize how the finished film will look while shooting just the physical elements.

All of this digital work required some mega-computing power behind the scenes as well. Shots containing a large number of intricate water-based effects could require weeks for Wētā’s computers to fully process. Rendering the finished film required millions of processor hours to complete. According to The New York Times, the total amount of data produced for “Avatar: The Way of Water” tops out at around 18.5 petabytes, compared to just 1 petabyte for the first movie.