Checking 30 Languages Without Speaking Them

Disney+ delivers audio and subtitles in more than 35 languages. Every dub and subtitle track carries a declared language, and a wrong one reaches customers unless someone catches it, but no QC technician speaks every language. I built an automated check that verifies the declared language before costly human QC begins.

Models

Open source: Silero VAD, Whisper, VoxLingua107, MMS-LID and Demucs for speech; fastText, langid and CLD3 for subtitles

My Role

Sole designer and developer, end to end, from audio preparation to the reviewer report

Why It Is Hard

• Dialogue is often buried under music and effects, and mix quality varies widely.
• One track can contain several languages, and dialogue intelligibility is often low.
• Dialects sit inside language families: Spain and Latin American Spanish, European and Quebec French.
• For subtitles, the declared tag and the filename can disagree with the text itself.

How It Works

1. Prepare the Audio

   Decode to 16 kHz, detect stereo or surround, and pick the cleanest source: mono as is, a stereo downmix, a left-center-right downmix for surround, or the center channel alone when the mix is still noisy.

2. Keep Only Speech

   Silero voice-activity detection drops music and silence before any language model runs.

3. Adapt the Window

   Analyze in 60-second windows. Where confidence falls below 75%, step down to 30, 15 and finally 10 seconds to isolate the uncertain stretches.

4. Escalate Through Models

   Whisper runs first and settles about 85% of windows. VoxLingua107 takes over when confidence is low, and MMS-LID gives a third opinion. When dialogue is buried in the mix, Demucs separates the vocals and the cascade runs again.

5. Refine Dialects

   MMS separates Norwegian, Chinese, Portuguese and Spanish variants. For French, transcripts from two models vote between European and Quebec French.

6. Decide and Report

   Each window scores duration × speech ratio × confidence, summed per language; the highest total wins, and a second language is reported only when it carries real weight. Reviewers get JSON flags, or an HTML report that plays back each window.

Subtitles follow the same idea on text: fastText first, langid for ambiguous cues, CLD3 for close language pairs, then a dialect pass and a check of the declared tag against the filename.

Designed to Run at Scale

• Heavy models run only on the minority of windows that need them, which keeps GPU needs small.
• Files process in parallel, and per-file seeding gives identical results in parallel or in sequence.
• A live monitor tracks progress, model usage and per-language accuracy during each run.

What It Will Change for QC

The check is not in production yet. Once it is, routine content will be validated automatically, so technicians can spend their time on premium titles and the edge cases that need human judgment. Language mismatches will be caught before they reach customers, and flagged spans will land in the same review tool as video and audio flags.

What I Would Tell Another Team

• Run the cheap model first and escalate only on low confidence. The same pattern works for speech and for text.
• Quote the false-flag rate next to the catch rate. A catch rate on its own is half a result.
• Ask people to verify flags, not to watch whole files.

What Is Next

Cross-checking dubs against subtitles on the same title. Each track can pass on its own while the pairing is wrong, such as a French dub with Spanish subtitles.

See Where This Sits in the Pipeline

Built at Disney. This page covers methods and aggregate results only.