45% of clips tagged 'lip-sync' were never checked for speech

By Alpha Lay · 2026-09-26

This is the least flattering piece about our own data.

We ran a round of speech detection (ASR) across the case library to answer something basic: does this video have anyone speaking in it? With that field you can separate talking-head content from pure-visual content, and you can evaluate things like lip-sync capability.

After the run we reconciled the results, and found that two dimensions that should corroborate each other disagree more than half the time.

Where the data comes from

• The speech field (asr): segmented detection over the audio track, producing four outcomes — speech segments detected (dialogue_detected), audio track present but no speech segments (no_segments), suspected recognition hallucination (suspect_hallucination), not run this round (not_run_this_round).
• The "lip-sync" technique tag (tt): human/semi-automatic annotation, meaning "the frame contains lip movement corresponding to sound."

The two are produced independently, which makes cross-validation meaningful.

Finding 1: For more than half the corpus, the question was never answered

55.3% of cases have no speech conclusion at all. And the gap is unevenly distributed — it is worst exactly where it matters most:

P0 is our own definition of "most important to understand first" (largely short-drama dialogue). Three quarters of it does not even know whether anyone is speaking.

Finding 2: Of clips tagged "lip-sync", 45% were never verified

This is the core of the piece. We took every case carrying the "lip-sync" tag (313 of them) and looked at its speech result:

Of 313 lip-sync cases, 140 (44.7%) had no speech data backing the tag when it was applied.

So where did those 140 tags come from? Only from looking at the picture — it looked like someone was talking. But "looks like speech" and "there is speech on the audio track" are two different claims: a shot of a mouth opening and closing can sit on a purely musical track, or on no track at all.

Finding 3: The reverse direction disagrees too — only 47% of detected speech carries the tag

Flipping the cross-tabulation:

More than half of the cases where speech detection says "someone is speaking" never received the lip-sync tag. Some of that is legitimate — voiceover, narration and singing can register as speech without involving lips. But at a scale of 132 cases, a real portion must be misses.

What you can do with this

If you want to evaluate lip-sync capability: the usable sample here is 119 cases (speech detected *and* lip-sync tagged), not 313. Using 313 as the denominator dilutes any conclusion.

If you are picking a model for talking-head content: note the 73.8% gap in the P0 tier. It means the best practices for talking-head content have barely been systematically organised in the public corpus. That gap is an opportunity, not a blocker.

If you annotate data yourself: this piece is a ready-made cautionary tale. Never let "looks like" stand in for "measured." 45% of our lip-sync tags lack measurement, and that is more than enough to invalidate the field.

Method and limitations

• The threshold for "speech detected" affects results: very short speech segments, music over voices, and non-native accents can all cause misses. We did not run human listening verification.
• suspect_hallucination has only 3 cases, far too few to support any claim about ASR accuracy.
• This does not establish that "models can't do lip-sync." The 50 no_segments cases could be mislabeling (visible lip movement with a silent track, i.e. a silent film) or a detection miss. This piece does not distinguish them.
• All figures use the 1,274 index records. After de-duplication: 374 never processed, 244 with speech detected, 301 audio-but-no-speech — proportions hold. Among the 224 de-duplicated lip-sync cases, 51 (22.8%) were never processed — lower, indicating the duplicated mirrors fall mainly inside the already-processed batch.

Reproduce it

Filter by speech result, or cross-reference against the lip-sync technique tag: https://cases.aishifu.shop/