Building an AI Dubbing Pipeline for Micro-Dramas

Dashverse is an AI-native storytelling platform. We use frameo.ai, our own AI video production studio, to produce micro-dramas and publish them on our consumer apps, DashReels and ShortFree. A micro-drama is a fast-paced, vertical video series of 60-100 episodes of 1-2 minutes each, about 2.5 hours in total.

A show is usually produced in one language, while our users are spread across many. At the pace we launch shows, professional dubbing quickly became the bottleneck. So we built an AI pipeline to fix that. It has since dubbed about 1,400 hours of micro-dramas across 16 languages with a turnaround time in days instead of weeks, and at a fraction of the cost.

We tried the existing AI dubbing products first, but none of them suited our needs. They handled regional Indian languages poorly, and their outputs felt more like robotic translations than proper "dubs." Dubbing, as we need it, is broader than a voiceover. It means reconstructing the screenplay, translating it with real cultural adaptation, and delivering it with voice acting that understands the scene.

Handling these required precise control over the whole process, which meant building the entire pipeline ourselves. Here's a quick look at it; the sections below go through it stage by stage.

Reconstructing the Screenplay

The obvious first step in automatic dubbing is getting a diarized transcript: who said what, and when. But this is neither trivial nor limited to simply getting a transcript from a speech-to-text (STT) model. It's in fact closer to reconstructing the original screenplay for three reasons:

• A good translation needs social and scene context, because language and grammar depend heavily on social hierarchy, relationships between the characters, and setting.
• Text-to-speech (TTS) generation needs accurate performance direction. Imagine a red-faced, crying character on screen voiced like a customer service agent.
• Voice casting needs to understand the character (their age, gender, role, and persona) so that the drama doesn't turn into unintentional comedy the second they open their mouth.

Labeling Who Speaks When

A correctly diarized script is essential for everything downstream. Transcription is the easy half of it. Most STT models can handle a show's 2.5 hours of audio. But diarization is where things fall apart.

Diarization degrades as the input grows, since the model has to track more voices and for longer. Micro-dramas make it harder still: they are packed with fast-paced storylines, tons of scenes, and a roster that often runs to 20-40 characters, many of them one-offs.

Blindly chunking the audio for diarization doesn't help either. A character's tone can change and settings can shift between chunks, yet the labels have to stay consistent across all of them.

Our initial attempts included experimenting with different models, tweaking their thresholds, running stem separation to isolate vocals, and using multimodal LLMs. But none of it alone was good enough for the accuracy this use case demands.

After a lot of trial and error, we ended up with an ensemble pipeline: signals from STT, voice activity detection, and the visuals are fed into a multimodal LLM along with the original video to get accurate speaker labels. But even context windows have limits, so the pipeline still processes the show in chunks, this time reconciling character identities across them using visual and contextual cues.

Reconciliation turned out to be an interesting challenge in its own right! A character might disappear for ten episodes and return in a new scene, wearing different clothes. The pipeline has to recognize them as the same person and keep their identity consistent.

As we'll see later, a human stays in the loop to guide the process and make minor fixes. During human review, only 5.2% of lines were relabeled. Treating the human-reviewed output as rough ground truth, we measured a diarization error rate (DER) of 8.2%.

Directing Each Line

Dubbing needs good voice acting. A line like "Why did you do that?" could be sobbed in heartbreak, murmured in dread, or screamed in pure anger. Thankfully, most modern TTS models offer some control over emotion and can add vocal expressions like laughter, cries, and sighs inline in the text. But in dubbing, the vocal performance is also tied to the expressions and energy of the actor on screen.

This means that every line in the transcript has to be "directed," with its emotional delivery and non-verbal vocalizations extracted from the video alongside the text. Again, a multimodal LLM handles this, given the individual video clip, its transcript, and some surrounding context.

Getting the right set of audio tags for the TTS also took some experimenting, because the models don't respond to all tags equally; some deliver noticeable emotions, while others barely make a dent. The rule of thumb we settled on: action-first tags like [whispers], [sobs], and [stammers] work much better than abstract descriptors like [condescendingly], [sarcastically], or [bitter].

Building the Character Roster

A speaker label like Speaker 1 isn't very useful by itself. All it says is which lines come from the same character. Dubbing a character faithfully requires knowing who that character actually is.

So as part of reconstructing the screenplay, the pipeline builds a show-wide character roster, with a detailed profile for every character. This includes demographic tags like gender and age group, story roles (male_lead, scheming_stepmother, bodyguard_1), aliases, and honorifics. This roster governs many downstream steps: determining pronouns during translation, matching vocal archetypes during voice casting, and keeping character identities consistent across dozens of episodes.

Setting the Localization Rules

Localization is what separates a dub from a literal translation. Think of the following examples:

• A million-dollar lottery literally translated as a ten-lakh-rupee lottery would drastically reduce the financial stakes, impacting the storyline.
• Translating an ancient Shaolin monastery as an ashram may make it more familiar to an Indian audience, but erases its Chinese Buddhist and martial-arts context.
• 오빠 (oppa) in Korean is highly context-dependent. It can mean an older brother, an older male friend, or a boyfriend. A wife suddenly calling her husband bhaiya would be catastrophic for the plot.

In professional dubbing, this is handled by building a glossary, a character bible, and a style guide. We borrowed that process, with a lot of help from our in-house writers and artists.

Our pipeline builds its own blueprint for adaptation using the transcript and character roster from the previous stage. It assigns each character an appropriately localized name and maps their aliases and honorifics. It also extracts all other glossary terms from the transcript (places, organizations, people, currencies, number units, and idioms) and maps each one to the target language in context.

Doing this up front changes what translation has to do. Instead of deciding on the fly what a particular word means in this scene, the LLM just looks it up. This cuts hallucinations, keeps terminology consistent across hours of translated dialogue, and makes verification easier.

Localization is where off-the-shelf LLMs start to show the limits of their understanding of different languages, especially regional Indian ones. The gaps become severe once the pipeline moves from building the blueprint to full translation.

Translating for the Right Register

With the localization blueprint built, translation would ideally be a single step: the blueprint and the screenplay go in, and the translation comes out. But Indian languages are still hard for off-the-shelf LLMs. And a language isn't a single, fixed system. It usually has different dialects and registers.

The register in particular has to shift depending on the setting of the show. For example, a period drama set in a royal court needs formal (shudh) Hindi, while one set on a modern college campus needs a much more colloquial register. Colloquial registers in Indian languages involve heavy code-mixing, usually with English, and getting that balance right is important too.

Further, some languages are deeply diglossic: Tamil, for example, has a sharp split between its spoken and written forms. And even with plenty of context, LLMs tend to default to very "bookish" or formal registers.

This is largely an artifact of their training data. In Common Crawl, Tamil, Telugu, Kannada, and Malayalam make up less than 0.05% each, while English sits at about 42% and Hindi at 0.2%.

Prompt optimizations can pull the LLM toward the right register in languages like Hindi, but it's less effective in Dravidian (South Indian) languages such as Tamil and Telugu. On top of that, off-the-shelf LLMs hallucinate more frequently and make spelling and grammatical mistakes when writing in native Dravidian scripts.

The solution was right in front of us. Over the past few years, our writers have created or adapted thousands of stories for our micro-drama apps, in exactly the regional languages and registers the shows need. That made fine-tuning LLMs for translation possible.

From these stories, we compiled a clean, well-labeled corpus of about 340,000 lines of dialogue across Dravidian languages, roughly 14 million tokens of target-language text. Each training example pairs the exact prompt our pipeline sends for translation with the human dub of the same stretch of dialogue, so the model learns to answer that prompt the way a dubbing writer would.

We used this corpus to fine-tune per-language, per-register translation models, with a dynamic routing layer on top that decides which model to invoke based on the register and vocabulary a show requires.

Voicing the Dub

Casting the Voices

The first step in voicing the dub is choosing the right voice for each character. While most TTS providers offer voice cloning, it consistently carries over heavy accents and introduces pronunciation mistakes, both of which get worse for regional Indian languages. So the pipeline sticks to casting from prebuilt voices.

TTS providers mostly tune their models and voice libraries for customer service and call-center voices; when they do branch into entertainment, it's for narration. But dubbing for drama demands line delivery that is natural, expressive, and adaptable across different character types and contexts.

So we manually curated about 3,000 voices from their libraries across every target language. Each voice is hand-labeled on eight axes, including age, gender, persona, and expressiveness. A casting algorithm then matches each character to the best-fit voice, weighing both how closely a voice resembles the original performer and how well it matches the character on those same eight attributes.

Fitting Lines to Their Slots

With the translated screenplay and voice casting in hand, dubbing may seem as simple as calling an API to generate the TTS clips. But there's one crucial constraint at the heart of dubbing:

The translated dialogue must fit within the same time slot as the original.

Translated lines often overshoot or undershoot their slot, and a perfect match is rare. For example, across a sample of a few Chinese-to-Hindi dubbed shows, the median first take was 1.45× the original slot. Most of this mismatch comes from the language pair itself. Chinese is very dense, while Hindi often takes relatively more words to say the same thing, so almost every line starts out too long.

Adjusting the speed of the audio buys a little room, but beyond a narrow range it starts to sound unnatural.

So fitting becomes a constrained optimization problem: rephrasing the line to fit the slot while preserving the original context, tone, emotion, and dramatic impact. A bad rephrase can undo all the work put into reconstructing and adapting the screenplay.

After rephrasing, the median accepted take came in at 1.02× the slot. Here are three takes of one line:

Mixing the Final Track

The dubbed dialogue tracks then have to be mastered and assembled into a production-ready show. This involves several audio-engineering steps: volume normalization, remixing the isolated background music and sound effects, and adaptive ducking. Background audio is separated from dialogue early in the pipeline; here it is mixed back in and "ducked" to a lower volume wherever someone speaks, so dialogue stays clearly audible.

This stage also generates the final SRTs, which are then transliterated into Roman script rather than the native script.

Keeping Humans in the Loop

After a few weeks of iterating and building proofs of concept, we realized that consistently getting a perfect dub straight out of the pipeline was nearly impossible. A single bad line can mess up the entire episode, whether it's a mislabeled character, a poor translation, or a flat delivery. Some calls also come down to pure creative judgment: landing the impact of a hook or cliffhanger, choosing the right register, or even picking the characters' names.

So every show goes through human review. Reviewers steer the translation, override casting choices, and fix weak lines and diarization errors. Most of the output survives that review: across about 140 hours of recent content, reviewers kept about 95% of speaker labels, 89% of translated lines, and 96% of casting choices as generated.

Even with a human in the loop, a show now goes from source video to finished dub in a few days instead of the few weeks professional dubbing took, and it costs an order of magnitude less.

Watch the Dubs in Action

Here are a few episodes of micro-dramas dubbed using the pipeline.

What's Ahead

Dubbing with AI turned out to be much harder than its one-line problem statement suggests. Getting here took months of experimentation, proofs of concept, feedback cycles, and architectural iterations.

The bigger outcome is what the project became. Once an LLM learns to reconstruct a screenplay from raw video, extracting the characters, dialogue, scenes, and arcs, that structure is useful for far more than dubbing.

The pipeline can now use the same structure to analyze dramatic arcs and detect high-tension hooks and cliffhangers, automatically cut trailers for marketing campaigns, and drive better recommendations across our consumer apps. What started as a dubbing pipeline quickly grew into a broader content intelligence track within the company.

As for dubbing itself, the ceiling is still the TTS models: their expressive range, their control over pitch, pace, and delivery, and their consistency. In Indian languages beyond Hindi, the ceiling is even lower, with non-native accents, frequent mispronunciations, and awkward code-mixing. The encouraging part is that regional voice models are getting better fast, with more TTS models training on authentic Indian speech. As they improve, the "AI" in AI-dubbed content will get closer to disappearing entirely.