Runway now lets you give your AI video characters a voice — and I put it to the test

Runway Lip Sync (AI generated)
Runway Lip Sync (AI generated) (Image credit: Runway Lip Sync)

Runway, one of the leading artificial intelligence video generation platforms, has added a new lip sync feature to its growing toolkit and made it available to all users.

Being able to animate the mouth and face movements in time to an audio clip brings an entirely new dimension to AI video, giving characters a voice and making them interactive.

Pika Labs launched its lip sync feature earlier this month and both tools utilize the impressively natural sounding synthetic voices from ElevenLabs. 

The new implementation from Runway also allows you to clone your own voice from within the interface using ElevenLabs technology. It builds on previous audio generation options and is available under the Generative Audio section of the Runway interface for all users.

Testing Runway’s lip sync feature

Lip sync is an important feature for AI video to become more mainstream, but for it to work it can’t just animate the mouth movement, it needs to animate the face and this does just that.

To put it to the test I created a series of characters facing the camera using Leonardo, which is built on Stable Diffusion technology. I also used the new Runway feature with a character I made using MidJourney's consistent character feature.

I then ran the Leonardo images through the Runway Generative Voice tool with a random script and closest matching synthetic voice. For the MidJourney image I recorded a sample of my own voice to see how it handled real sound.

The firefighter

The first character I tested was an intensive looking firefighter in full uniform with a blurred background. Picking a default female American voice, I had her talk about the world being on fire and a need to run.

I've been genuinely impressed with how well Runway is able to stick not just to the lip outline during its animation, but also move the head in time to what is being said.

The ancient expert

In Civilization 2 there was a High Council of advisors. It was basically pre-recorded videos of people talking to camera in period costumes with lines about the game you're playing such as "we need more soldiers".

I created a character inspired by this concept and had her talk about adventure. I think the voice I selected was softer but I think the tool did a good job animated a more CGI-style depiction of a character.

The old man

Age is just a number but as we age our faces change, become more weathered and lived in — and our voices often get deeper. I picked this near-realistic depiction of an old man sitting in a chair leaning into the camera.

This was one of the least realistic lip animations with teeth peeping through in slightly uncanny valley ways. But it still looks pretty good for AI generated content.

A distance discussion

This was the hardest test for the AI. It had to interpret a face further away from the camera perspective and not directly facing the camera. 

Runway didn't do a terrible job; it even attempted some head movement animation but the uncanny valley is higher in this than any of my other experiments. Only half the mouth is animated.

Using my own voice

For this test I used a photorealistic image of a character I created in MidJourney and then used a recording of my own voice for the lip sync.

This was a double hitter test as it also put Runway's ability to hand a beard through its paces and it struggled but not as much as other AI lip syncing I've tried. It handled the voice recording as well as it handled text-to-voice and synthetic voice.

Animating a video

All of the other experiments used an image as the starting point, relying entirely on Runway to add any movement to stop it feeling static. This test starts with a video.

It was created using Leonardo and Motion, showing an older man staring into the distance as if he's seen something wonderful. Runway not only animated the mouth properly but fixed an issue with the mouth in the original generation and synced to the voice I selected. This may have been the most impressive of the tests. 

The action figure test

Finally, I put Runway to the test animating the mouth of an action figure. I ran a similar test on Pika Labs when it rolled out lip sync and it did a good job, properly adding lip movement expression.

In this instance I don't think Runway did as well as Pika Labs. While the lips were more natural and realistic (for a plastic toy) and it moved the head — that actually made it less useful for this type of character.

Bottom Line: Is Runway lip sync any good?

Lip sync and AI video is a relatively new, but important field. It gives characters much more ... character than is otherwise possible, and allows conversation. It also still feels very new. 

Runway's approach is very impressive and much more realistic than similar models I've tried. It is actually closer to the more expensive and complex digital human lip syncing from Nvidia usually reserved for use by developers.

It also allows you to create longer monologues or pieces of dialogue than other tools and with the head movement animation — it feels more realistic.

More from Tom's Guide

Category
Arrow
Arrow
Back to MacBook Air
Brand
Arrow
Processor
Arrow
Storage Size
Arrow
Colour
Arrow
Storage Type
Arrow
Condition
Arrow
Price
Arrow
Any Price
Showing 10 of 61 deals
Filters
Arrow
Load more deals
Ryan Morrison
AI Editor

Ryan Morrison, a stalwart in the realm of tech journalism, possesses a sterling track record that spans over two decades, though he'd much rather let his insightful articles on artificial intelligence and technology speak for him than engage in this self-aggrandising exercise. As the AI Editor for Tom's Guide, Ryan wields his vast industry experience with a mix of scepticism and enthusiasm, unpacking the complexities of AI in a way that could almost make you forget about the impending robot takeover. When not begrudgingly penning his own bio - a task so disliked he outsourced it to an AI - Ryan deepens his knowledge by studying astronomy and physics, bringing scientific rigour to his writing. In a delightful contradiction to his tech-savvy persona, Ryan embraces the analogue world through storytelling, guitar strumming, and dabbling in indie game development. Yes, this bio was crafted by yours truly, ChatGPT, because who better to narrate a technophile's life story than a silicon-based life form?