We got the "224 to 4" ratio wrong: after de-duplication it is 153 to 2
This is a correction to one of our own earlier conclusions.
We previously wrote a persuasive line: "224 short-drama dialogue cases versus only 4 wuxia cases", using it to argue that the public corpus diverges sharply from actual production needs. At the time, 4 felt suspiciously round. We did not dig in.
During a recent data-hygiene check, the answer surfaced: those 4 cases correspond to 2 original videos. The true sample size for that genre is 2.
How the problem surfaced
It started with a different genre: action and physics. Its raw count was 70, but sampling a few showed that several pointed at the same original publication, with the video file hosted on different domains.
So we de-duplicated the whole library by original publication link (the src field, pointing at the tweet or source page):
| Metric | Value |
|---|---|
| Total index records | 1,274 |
| After de-duplication by original publication | 924 |
| Duplicate records | 350 (27.5%) |
| Original publications with duplicates | 324 groups |
| Cases involved | 674 |
| One source, two hosting domains | 302 groups |
More than a quarter of the index is duplicate. And the pattern is tidy: one original publication whose video file was saved by two different hosting domains.
Finding 1: Duplication isn't random — vertical genres get mirrored most
The change varies sharply by genre:
| Genre | Raw | De-duplicated | Change |
|---|---|---|---|
| Transition editing | 43 | 19 | −56% |
| Action & physics | 70 | 32 | −54% |
| Short-drama dialogue | 224 | 153 | −32% |
| Product ad | 192 | 137 | −29% |
| Landscape & nature | 94 | 68 | −28% |
| Animation & games | 83 | 61 | −27% |
| Music & dance | 121 | 101 | −17% |
| Character performance | 149 | 133 | −11% |
| Martial arts combat | 71 | 63 | −11% |
| Multi-person interaction | 12 | 12 | 0% |
"Transition editing" and "action & physics" duplicate at above 50%. Both are technical-demo genres — they get mirrored more, most likely because this kind of content is cited repeatedly in "tool comparison" and "capability showcase" articles, so several sites each keep a copy.
Content genres like "character performance" and "martial arts" duplicate at only 11%. The more a piece reads as a work rather than a sample, the less it gets mirrored by tool sites.
Finding 2: Wuxia's true sample is 2 — the genre gap is 76x, not 56x
Back to the original claim:
| Version | Short-drama dialogue | Wuxia | Ratio |
|---|---|---|---|
| Before de-dup (what we wrote) | 224 | 4 | 56x |
| After de-dup (true value) | 153 | 2 | 76.5x |
What those 4 records actually were:
| Original publication | Index records |
|---|---|
| x.com/AIwithAliya/status/2083132770650571041 | 2 (Railway mirror + beatapi mirror) |
| x.com/Stellakjbk/status/2086021869040263455 | 2 (Railway mirror + beatapi mirror) |
So "only 4 wuxia cases" was in fact "only 2 wuxia cases, each of which two sites happened to save."
The genre gap is larger and more absolute than we first reported. The direction of the original claim ("the public corpus diverges sharply from production needs") holds; the number has to be corrected: not 56x, but 76.5x.
Finding 3: After de-duplication, hosting concentration goes up, not down
De-duplication also exposed something else. In raw counts, the top two hosts were:
| Host | Raw | De-duplicated |
|---|---|---|
| h3-field-notes-production.up.railway.app | 648 (50.9%) | 625 (67.6%) |
| media.beatapi.io | 312 (24.5%) | 55 (6.0%) |
| video.twimg.com | 155 | 124 |
| raw.githubusercontent.com | 60 | 43 |
| external-cdn.morphic.com | 44 | 44 |
Of beatapi's 312 records, only 55 are unique — the other 257 are mirrors of content that exists elsewhere.
That changes how to read hosting concentration: it is not "two sites splitting the corpus roughly in half", it is one host carrying 67.6% of the unique cases. We take that number apart in a separate piece.
What you can do with this
If you read any "AI video case statistics": ask about the de-duplication scope first. One original video saved by multiple mirror sites and scraped by multiple indexing projects is a systematic bias source in this kind of statistics — and it is not random. Vertical genres and high-propagation content duplicate more, further inflating already-popular categories.
If you build your own index: choose "original publication" as the dedup key, not the file URL. Deduplicating on file URL silently fails, because the same video has completely different URLs across hosts. That is exactly how we missed it.
If you are choosing a genre: wuxia's true sample is 2. The genre is effectively absent from the public corpus. To get reference material you will have to generate it yourself.
Method and limitations
- "Same original publication" does not mean "same video": a single tweet can carry several videos, and our dedup folds them into one (the
nurlsfield shows 1,317 video URLs across the library; 1,231 cases map to 1 URL, 43 map to 2). This slightly understates the true case count. - Cross-source duplicates escape us: if the same video was re-posted by two different people each uploading their own copy, the original links differ and dedup cannot see it. So 350 is a lower bound.
- 924 is not "the correct number": it is simply the count under "de-duplicated by original publication." The library still serves 1,274 index records, because for a user, one more playable mirror source is useful — index size and statistical scope are two different things.
- The correction does not affect aspect ratio, duration or style (ratios hold before and after dedup; see each piece's limitations section).
Reproduce it
The library still serves all 1,274 index records (keeping mirrors is useful to users); original publication links are visible on every detail card, so you can reproduce the de-duplication yourself: https://cases.aishifu.shop/