With the recent announcement of their Gen-3 Alpha tools, Runway is set to take AI video creation to the next level. By the end of this guide, you'll have a clear understanding of how to use Runway's lip-syncing tool and be able to decide if it's the right solution for your video creation needs.
As one of the early developers of the groundbreaking Stable Diffusion image generation platform, Runway brings a wealth of experience to all aspects of AI-powered image and video creation. One of Runway's most accessible and popular features is its audio to video lip syncing tool. This technology allows users to create realistic talking head videos by matching lip movements to audio input. Whether you're a content creator, educator, or business professional, this tool offers a range of applications for enhancing your video projects.
1. Log in
Log into Runway. If it's your first time using the AI platform, you can sign up using your Gmail account.
2. Navigating the interface
The video lip-sync function is under the audio menu on the Runway account dashboard alongside 3D tools, and image and video generation. Select your preferred voice from the options.
What's nice about the user interface is the liberal use of tutorial material at every stage, which is reassuring for beginners who are starting out on the AI media journey.
3. Input your prompt
The lip-sync page itself is comfortingly simple and easy to use. The top left is the space for user inputs. This can be via a text prompt, an uploaded audio file or a tool for recording audio on the fly.
If you upgrade to the Pro plan for $35 a month, and you can even record your own custom voice to use with an avatar.
4. Choose your voice
I decided to upload an AI-generated avatar I created earlier, and combine it with a short script to test out the accuracy of the lip movements.
There's a great range of voices to choose from, which probably reflects Runway's history of working closely with the movie industry, so there's something for everyone.
5. Generate your video
Once you've done the audition and selection of voices, it's just a matter of uploading a script (or pre-recorded audio clip) and pressing the Generate button.
The free plan generates at a slower pace than the paid plans, so you have to wait for a few minutes in a queue before your request is actioned. The whole process takes around 3 to 4 minutes depending on the length of the video, and the server load at the time.
6. Understanding credit use
Text to Speech uses 1 credit for every 50 text characters converted to speech, and lip sync consumes 5 credits per 1 sec of audio.
The total video combines both costs, which in effect meant that my 38-second video used around 250 credits. Each credit is worth $0.01 USD. Standard Plan users get 625 credits each month for their $15 subscription.
The results...
Skip to 3.25 to watch the final creation below.
After the video has finished generating, you can preview the results on the right-hand side of the web page. Free users have a watermark attached to every video, but at least it's not too ugly.
The actual video results are also really good, even using an uploaded image, which I found surprising. The lips move accurately during speech, and there's even some facial movements to match.
The overall effect is not perfect, there's still a bit of 'uncanny valley' in action, but it's very good for general business needs. There are a number of options available once the video is created, including download, re-generation, editing the script and other elements.
It's a fast and painless process all around, and the results are really top-notch as might be expected from a company with this pedigree.
There are many similarities between Runway and Pika Labs, another prominent player in the AI video generation space. In fact, Pika Labs has a similar lip sync feature, which also uses the natural-sounding voices from ElevenLabs. There are many AI video generators out there, with the best ones yielding impressive results.
