Our own backlog: 98.9% of our highest-priority tier has never been reviewed

By Alpha Lay · 2026-09-26

The preceding pieces covered what the data says. This one covers how unreliable the data itself is — including the part we caused.

Where the data comes from

Every case carries three status fields:

• status: searchable-pending-review / metadata conflict / composite comparison sample
• flags: specific quality problems on that record
• Per-dimension "unreviewed" markers: 225 technique, 421 style, 22 genre

Finding 1: 95.1% of records are in a "pending review" state

Not one case is in a "fully reviewed" state. The schema has no such value — we set the goal at "searchable" from day one, and review became permanent debt.

Finding 2: 707 records only ever reached "automatic candidate", and the two flags are the same set

Quality flags:

The first two have identical counts, and they are the same 707 cases. Not a coincidence: it means those 707 went through one processing path — cloud ffmpeg technical extraction only, with all tags from automatic candidates, and no human review whatsoever.

What about the other 567?

Among those 567, 454 come from provenance "verbatim from the original publisher" — tags taken directly from source text, more reliable than automatic candidates, but still never semantically reviewed.

Finding 3: The tier we labelled highest-priority is the least reviewed

This is the admission we owe most. The priority field splits cases into three tiers; P0 is defined as "current short drama" — by our own stated intent, the tier most important to understand, and most important to review first:

The tier we called most important has the worst review rate. This is clearly a process design failure: priority was assigned at collection time, review never got scheduled, so "priority" ended up affecting exactly one thing — the order in which records got indexed — and nothing about quality.

And within P0, 194 cases (73.8%) never even had speech detection run (see the speech piece). The priority field, in our hands so far, has functioned as a label, not as allocation of effort.

Finding 4: A few small but telling quality flags

"Shot-by-shot coverage under 90%" is only 5 cases, but all of them are long videos — exactly the category most in need of shot-level analysis. That coincidence is worth heeding: the more complex the long video, the lower our analysis coverage.

What you can do with this

If you cite any of our earlier pieces: treat this as a standing disclosure. The direction of those conclusions is sound (they rely on ori, dur and other objective measurements), but every classification conclusion — genre, technique, style — shares a denominator that is 95% unreviewed.

If you are designing your own annotation pipeline: this is a complete cautionary case. The core lesson: "pending review" must be a bounded queue, never a default state. We turned review into "later, when there is time," and the result is 1,212 records sitting in a queue with the highest priority last.

If you evaluate any auto-annotated dataset: three questions are enough. (1) What share is automatic candidates? (2) How much has been human-reviewed? (3) How was review prioritised? Our answers: 55.5%, under 5%, and wrongly.

Method and limitations

• This piece is itself meta: it measures the quality of our records, not the quality of the videos, so it carries no additional sampling bias — it is simply an accounting.
• "Review" is not strictly defined. We never wrote down what "passing review" means. That is itself one of the defects, now on the improvement list.
• status has only three values and none means "done" — a problem baked in at schema design time.
• All figures use the 1,274 index records.

Reproduce it

Each case's status and quality flags are public in its detail view: https://cases.aishifu.shop/