Watching a movie in a foreign language can be fun, but few people enjoy the distraction of reading the closed caption. Dubbing a film into the native language may be better. However, the new dialogue does not match actors’ mouth movements, creating a dissatisfactory experience. Sometimes, the script is modified to better match mouth movements. Yet this may hurt the accurate and nuanced understanding of the dialogue. Flawless AI, a company, created a piece of software called TrueSync to address this issue. The neural-network-based technologies learn from an actor’s performance in the movie and slightly modify the mouth movements to match the translated dialogue. The technologies enable perfectly lip-synced visualizations in various languages. Importantly, the AI system captures the nuance and emotions of the original footage to enable authentic visual translation.
Summary
Purpose
In translating videos, match people’s mouth movements with translated conversations
Idea
Use AI to learn mouth movements and accordingly revise the visual
Further Possibilities
1. add accent to movies with modified mouth movements
2. Fully automated translation of movies
3. make new movies or audio work with AI-based learning of previous works of actors
4. Use AI to modify all digital arts
5. Use similar techniques for analyzing movies’ emotions
Questions
1. What might be all the artistic objects that can be modified by AI?
2. How else might AI create derivative work?
3. What are the drawbacks of using AI to modify art?
