Out-of-sync audio in hundreds of videos? Build a checker

The problem, in your words

You have a pile of recorded interviews, video calls and AI-made clips. Some of them have lips that do not match the voice. People keep asking how to find these without watching every file.

Nothing is wrong with you or your footage. A clip that is a fifth of a second off is easy to miss and still feels odd to a viewer. Scrubbing hundreds of files by eye is not realistic. So we built a checker, and we run it. These are our lessons.

What a good solution does

A good checker does five things, and it does them without you watching.

First, it takes a folder and walks every video in it. Second, it samples several moments in each file, not one. Third, it gives each file a plain verdict: pass, drift (the gap grows over time), step (the gap jumps), review, or unmeasurable. Fourth, it writes an HTML report and a spreadsheet you can sort. Fifth, it remembers what it has done, so a run of 500 files can stop and start again.

We built two pieces that share one command. The first is the analyzer. The second is a video editor on Remotion, which makes video from React code1. The editor runs the checker as a gate on every render, so a bad file never leaves quietly.

What you need before you start

• A computer with Node 16 or newer for Remotion6, Python 3.11 or newer, and ffmpeg.
• Claude Code, an AI coding assistant that builds features and runs commands for you5. See /start-here/ for a safe setup.
• A few gigabytes of free disk for the sync model.
• A folder of sample videos, including one you already know is off.

Whisper, a speech recognition model, can give you word timings if you want the editing side3. It needs ffmpeg installed too3. If you have no terminal habits yet, start with /start-here/ and come back.

The playbook

Work in an empty folder, and let Claude Code ask before each command. Go one phase at a time.

1. 1

   Scaffold the tool.

   Ask Claude Code for a Python package with one command, sync <folder>. It walks the folder, pulls the audio out and samples several time windows per video. Ffprobe, which reads facts about each stream4, helps here.

2. 2

   Add a lip-sync model for faces.

   Pick a published lip-sync network. Pin its weights by checksum, and keep its Python environment outside any cloud-synced folder.

3. 3

   Add a fallback for faces-free clips.

   Use audio-to-video cross-correlation, which compares sound spikes with picture changes. Trust it only when two or more windows agree. Otherwise say "unmeasurable".

4. 4

   Write the verdicts.

   Pass, drift, step, review, unmeasurable. Save results as JSON lines, plus the HTML report and a CSV.

5. 5

   Make it resumable.

   Keep a small cache per file so you can stop a long run and restart it.

6. 6

   Test with shifts you made yourself.

   Take a good clip. Delay its audio by known amounts. Confirm the checker reads them back within a few milliseconds.

7. 7

   Build the editor side, if you want it.

   Describe each cut in one small file, in seconds, and keep the frame math in whole frames. In Remotion, a video's first frame is 0 and its last is durationInFrames minus 11. Put the duration math in one function.

8. 8

   Add word timings.

   A transcript with a time for every word lets you delete a line and cut those frames.

9. 9

   Chain the stages.

   A recipe file can run analyze, cut, captions, render and check, and write a lock file so reruns match.

10. 10

    Finish with a loudness pass.

    Do it after the render, and let the sync checker gate every render.

That is the whole build. Nice work, even if you only get through step 6 today.

What we got wrong the first time

We averaged two checks. That was worse than gating them. On real talking heads, cross-correlation and the lip model disagreed by hundreds of milliseconds, because lips lead the voice by a changing amount. Now faces go to the lip model, and the second check only counts on sharp events.

Our checker also looked too narrowly at first. In one batch, a remote-recording tool's combined export had the host's video drifting against the audio by about 10 to 14 ms per minute, up to about a second. Readings past our search window were invisible, so the problem hid. For two-person calls, search wide, and check each face on its own.

A few more lessons, quickly:

• Call recordings carried a different steady offset per person. AI avatar clips ran a steady amount late. One global fix cannot work.
• The best repair used each person's raw audio track. We aligned the tracks to the edit and shifted each voice by its own measured curve. All 33 files in that batch passed again.
• Padding an odd number of frames at 25 fps biased a reading by 20 ms. Put cuts between speakers, not only at the quietest moment.
• Gain applied inside the render clipped the audio. Loudness belongs in a two-pass step after the render, with a true-peak limiter.
• Before you blame the checker, check the source tracks. In one case a guest could not be understood because the edit used a poor call-audio copy, not the person's own microphone. In another, a host on speakers had the guest's voice in their mic.

What it costs to run

As of November 2026, Remotion is free for individuals and companies up to three people2. Larger companies need a license, so read its terms first. The sync model, ffmpeg and a local Whisper are free. Our transcript-based cutter calls an AI model, and we used a Claude subscription through the command line rather than a paid API key. Disk is the real cost. Our model environment took about 1 GB.

Keep it safe

Footage and transcripts are personal data. Keep the outputs next to the footage, and keep them out of public repositories. Never overwrite an original. If you pull ad-platform numbers into the editor, use a read-only token, and type it only into the tool's own settings, never into a chat. Download model weights from their official page and check them against a published checksum, so you are never running a file nobody verified.

One last habit: give Claude Code a plain, empty folder to work in. Keep its approval mode on, so it asks before it runs anything at all.