That’s What He Said? A.I. Software Creates “Deepfake” German Speaking Robert De Niro

By B.N. Frank

A 2019 study revealed that 82% of Americans believed Artificial Intelligence (AI) technology is more hurtful than helpful. It’s been condemned for replacing human jobs (see 1, 2) and being used for unethical, harmful, and downright creepy applications (see 1, 2, 3, 4, 5). Nevertheless, it continues to be utilized by businesses and installed in “Smart Cities”.

AI can be used to create “Deepfakes” which can and do cause a great deal of confusion (see 1, 2, 3).

More from Wired:

This AI makes Robert De Niro perform lines in flawless German

Technology related to deepfakes helps match facial movements to dialogue.

You talkin’ to me … in German?

New deepfake technology allows Robert De Niro to deliver his famous line from Taxi Driver in flawless German—with realistic lip movements and facial expressions. The AI software manipulates an actor’s lips and facial expressions to make them convincingly match the speech of someone speaking the same lines in a different language. The artificial-intelligence-based tech could reshape the movie industry, in both alluring and troubling ways.

The technology is related to deepfaking, which uses AI to paste one person’s face onto someone else. It promises to allow directors to effectively reshoot movies in different languages, making foreign versions less jarring for audiences and more faithful to the original. But the power to automatically alter an actor’s face so easily might also prove controversial if not used carefully.

The AI dubbing technology was developed by Flawless, a UK company cofounded by the director Scott Mann, who says he became tired of seeing poor foreign dubbing in his films.

After watching a foreign version of his most recent movie, Heist, which stars De Niro, Mann says he was appalled by how the dubbing ruined carefully crafted scenes. (He declines to specify the language.) Dubbing sometimes involves changing dialog significantly, Mann says, in an effort to make it more closely match an actor’s lip movements. “I remember just being devastated,” he says. “You make a small change in a word or a performance, it can have a large change on a character in the story beat, and in turn on the film.”

Mann began researching academic work related to deepfakes, which led him to a project involving AI dubbing led by Christian Theobalt, a professor at the Max Planck Institute for Informatics in Germany. The work is more sophisticated than a conventional deepfake. It involves capturing the facial expressions and movements of an actor in a scene as well as someone speaking the same lines in another language. This information is then combined to create a 3D model that merges the actor’s face and head with the lip movements of the dubber. Finally, the result is digitally stitched onto the actor in a scene.

Flawless drew inspiration from Theobalt’s project for its product. Mann says the company is in discussions with studios about creating foreign versions of several movies. Other demo clips from Flawless show Jack Nicholson and Tom Cruise delivering famous lines from A Few Good Men in French and Tom Hanks speaking Spanish and Japanese in scenes from Forrest Gump.

“It’s going to be invisible pretty soon,” says Mann. “People will be watching something and they won’t realize it was originally shot in French or whatever.”

Read full article and see video