How does lip sync work in AI video generators?
AI lip sync works in one of two ways: either you supply the audio and a model reshapes the mouth of an image or video to match each sound, or the video model generates the speech and the picture together from your text, so the mouth and voice come out of the same render. A third variant feeds a song as an audio reference so the character sings or dances to it.
The three methods
| Method | You provide | You control the voice? | Typical use |
|---|---|---|---|
| Audio-driven lip sync | A face (photo or video) and a speech file | Yes, it is your audio | Talking photos, dubbing, avatars |
| Native audio generation | A text prompt with the lines in quotes | Only by description | Character scenes with dialogue |
| Audio reference | A song or voice clip plus a prompt | Partly; the model may re-sing it | Dance and singing clips |
1. Audio-driven: your audio moves the mouth
The model listens to your audio, works out which mouth shape each sound needs (open for "ah", closed for "m" and "b", rounded for "oo"), and redraws the lower face frame by frame while keeping the rest of the image. Better models also move the head, brows and eyes with the rhythm of the speech, which is what stops the result looking like a puppet.
- Kling's Avatar 2.0 takes an avatar image, a speech audio file and an optional prompt[1]. Its separate lip sync tool is priced by video length, with 5 seconds costing 5 credits[2].
- Hedra describes Character 3 as an audio-to-video model for talking and singing video[5].
- Dreamina lists lip sync audio generations among its paid-plan benefits[6].
This is the method to use when the exact words and voice matter: a voice-over you recorded, a translated dub, a podcast clip. Accuracy depends mostly on the inputs: a clear, unobstructed mouth and clean audio without music under the voice.
2. Native audio: the model speaks your lines
Newer video models generate sound with the picture. You write the dialogue into the prompt and the model produces the voice, the room sound and the matching mouth movement in one pass. ByteDance announced Seedance 1.5 pro in December 2025 as its first Seedance model with joint audio-visual generation[7], and BytePlus says Seedance 2.5 supports 11 languages for generated audio[9]. Kling says Kling 4.0 improves lip sync and adds multilingual dialogue[3], and that Kling 3.0 can give lines to several characters in one scene with matched voices[4].
What we learned rendering our own Seedance videos:
- Put every spoken line in quotes, attached to a time range:
0-4s: She says: "Reading my hate mail. Live.". Lines in quotes come out lip-synced; paraphrased dialogue gets mumbled or skipped. - Describe the voice in a few words, such as "posh English drawl, with matching mouth movement". You steer the voice; you do not choose it exactly.
- For narrated videos where nobody on screen should talk, write "Nobody speaks; mouths closed or smiling." Otherwise mouths move under the voice-over.
- Keep lines short. In a 5 to 15 second clip, one or two short sentences per character land reliably.
3. Audio reference: singing and dancing to a track
Some models accept a song as an input. Seedance 2.0 takes up to 3 audio clips per generation[8], and BytePlus says Seedance 2.5 accepts audio-only references while the 2.0 series needs an image or video alongside the audio[10]. In our dance videos the model re-sang the song rather than passing it through: it ran about 0.3 to 0.4 seconds late, and some takes kept only the dialogue. The fix was to lay the original track back over the picture, delayed 0.35 seconds, and keep the model's audio only where a character speaks. Our Seedance prompting guide has the full method.
Which method should you use?
- Your own voice or an exact script: audio-driven. See how to make a photo talk.
- A character saying a line inside a scene: native audio. See how to make an AI character video.
- A character singing or dancing to a song: audio reference, then re-lay the original track in editing. See can Seedance dance to my song.
Where Cast fits
- A character who says a short line inside a 5 to 15 second scene: Cast's videos run on Seedance 2.0 Fast with native audio, and quoted lines come out lip-synced.
- Trying native-audio dialogue without writing the full prompt: Cast expands one line into a timed prompt with the spoken line in it.
Where it does not:
- Lip sync to your own uploaded voice or song. Cast takes no audio upload.
- Dubbing an existing video, or making a photo you upload talk.
- Exact control over the voice; the model generates it from a description.
People also ask
- Do I need to upload audio for AI lip sync?
- Not with native-audio video models, which speak the lines you write in the prompt. You do with audio-driven tools: Kling's Avatar 2.0, for example, takes a speech audio file with the image[1].
- Does Seedance do lip sync?
- Yes. In our renders on both Seedance 2.0 Fast and 2.5, dialogue written in quotes comes out with matching mouth movement, generated in the same pass as the picture.
- How accurate is AI lip sync?
- Good enough for social clips when the mouth is visible and the line is short. It slips on long monologues, fast speech, profile angles and faces partly covered by hands or hair.
- Can I make an AI character sing?
- Yes, two ways: an audio-driven model that supports singing, such as Hedra's Character 3[5], or a video model that takes the song as an audio reference. Expect to re-lay the original track in editing for clean audio.
Sources
Claims about other companies come from their own pages, read on the date shown. Products and prices change; check the link for the current version.
- Kling AI: Avatar 2.0 user guide (read )
- Kling AI: lip sync guide (read )
- Kling AI: AI video generator (read )
- Kling AI: Kling Video 3.0 (read )
- Hedra: models (read )
- Dreamina: pricing (read )
- ByteDance Seed: Seedance 1.5 pro launch (read )
- ByteDance Seed: Seedance 2.0 launch (read )
- BytePlus ModelArk: Seedance 2.5 (read )
- BytePlus ModelArk: Seedance 2.5 (audio reference) (read )
Related answers
- How do I make a photo talk with AI?
- Can Seedance make a character dance to my song?
- How do I write a good Seedance prompt?
- Do I need a talking avatar or an AI character video?
- What is Seedance?
- What is the best free talking photo AI?
Part of the Cast answers library (Talking photos and avatars).
Updated