Sony Patents Next-Gen Lip Sync System To Match Different Translations With NPC Mouth Movements

Expert Verified By

This Could Change Lip-Syncing As We Know It In Games!

Story Highlight
  • A new Sony patent discusses a lip-sync system for game NPCs to match various translations easily with mouth movements.
  • The evaluation and fixing system can detect a similarity score for developers to ensure the character animations match all languages.
  • The system can also fix the duration of translated sounds to align them with the original dialogue’s timing, resolving sync issues.

Gaming studios are competing to secure the wildest patent ideas after huge innovations in AI and technology over the last decade. Now, a new Sony patent may revolutionize game development as we know it. 

We have discovered a new Sony patent that will shorten the game development time by automating a highly tedious process for developers. It discusses a next-gen lip-sync system for game NPCs to synchronize translations easily with mouth movements. 

The patent discusses a system that can evaluate how well NPCs’ mouth movements hold up across all the available languages and pass a similarity score to developers. It also includes automated solutions to fix synchronization problems.

Based on the shape of the character’s mouth corresponding to each translated phoneme included in the translated phoneme sequence, a similarity [Score] indicating the similarity of the character’s mouth movements may be generated.

Why it matters: Sony’s new patent could dramatically speed up game development while making NPC animations look natural in any language.

The image shows the original text, the translated sentence, the phoneme sequence before translation, and the phoneme sequence generated after transition.
The image shows the original text, the translated sentence, the phoneme sequence before translation, and the phoneme sequence generated after transition.

The patent, ‘TRANSLATION LANGUAGE EVALUATION DEVICE’ mentions that the system can adjust the duration of translated speech and the movement of NPCs’ mouths to better align with the original dialogue’s timing, which can automatically fix the syncing issues.

Sony argues that when NPCs’ mouths don’t match the translated dialogue, it turns off gamers by hurting immersion. Manually fixing these redundant issues can take a long time in game development.

[When] the sound of the translated text does not correspond to the movement of the character’s mouth, [Then] the player of the game or the viewer of the video work may feel uncomfortable.

The figure focuses on a point where the vocal periods overlap.

Overall, if Sony ever implements the system outside the patent, it might significantly shorten game development time.

Sony has secured many innovative ideas over the years, including one to create solar-powered controllers that charge in the presence of light and a unique custom recap system that creates highlight reels of missed gameplay events if the player stops paying attention.

Do you think Sony will utilize this lip sync evaluation and fixing system to shorten the time it takes to synchronize NPCs’ mouth movements? Let us know your thoughts in the comments below, or join the discussion on the Tech4Gamers forum.

Was our article helpful? 👨‍💻

Thank you! Please share your positive feedback. 🔋

How could we improve this post? Please Help us. 😔

Gear Up For Latest News

Get exclusive gaming & tech news before it drops. Sign up today!

Join Our Community

Still having issues? Join the Tech4Gamers Forum for expert help and community support!

Latest News

Join Our Community

104,000FansLike
32,122FollowersFollow

Trending

Nvidia Patents To Predict And Allocate CPU & GPU Resources In Advance To Improve Gaming Performance

A new Nvidia patent wants to predict when and how many resources CPU and GPU need and allocate them in advance to improve system performance.

These Components Should Be Avoided When Building A Budget PC

There are a lot of PC components that, if bought incorrectly, can truly ruin your PC build.

Why Game Budgets Continue To Soar With Each Generation

Costs in the gaming industry are up across the board, and this hobby will only become more expensive as the days go by.

Snag This Intel Core Ultra 7 Desktop 265K Processor For The Lowest Price Ever On Amazon

The popular Intel Core Ultra 7 265K processor is now selling at a staggering 36% off on Amazon in a new limited-time sale.

Why I Think Gaming Peripherals Deserve a Second Life

Old gaming gear still has life left in it. Here’s why I think repairing, modding, or passing on your peripherals beats chasing every new release.