Prove each check can fail before trusting its pass, then watch the actual film

In plain words

The problem: A check can pass for the wrong reason. A black-and-white check passed an image that still had colour in it, and a caption check missed text already printed on the source footage. A green result like that is worse than none.
What I did: Before paying for large test runs, every check must first reject a deliberately broken input. A human verdict is recorded against the exact exported film, and a revised cut needs its own review.
What happened: 29 of 29 checks proved they can fail. The repaired black-and-white check now rejects the coloured control. On a NASA-footage run, looking at the frames caught oversized graphics the AI reviewer had passed.

My contribution

I set the acceptance and spend-gating rules and defined the human review criteria.

A passing check can measure the wrong thing: a black-and-white check passed an image that still contained colour, and a caption check missed text already burned into the source. I put two conditions on accepting results: proof that each check can detect the fault it exists for, and a quality verdict tied to the actual film being delivered.

What the assistant sees

Checks the agent can read

The export reports its review1

"qc": { "checks": ["caption_sync", "dead_air", "hook"],
  "findings": [{ "id": "caption_sync", "severity": "defect",
    "message": "Captions are out of sync with the speech by about 1.1 s." }] }

And every frame-level check2

{ "id": "luma_defects", "status": "defect",
  "evidence": { "unexplainedDips": [{ "startMs": 0, "endMs": 1200, "minYavg": 35 }],
    "medianYavg": 115.7, "coverage": { "mode": "full" } } }

1A finding the agent has to show; acceptance of the exact film stays a separate decision Decision 2
2Each check carries its evidence and coverage, so a reader can tell a real defect from an intended one Decision 1

Start with a known failureAn ungraded, coloured image

A useful black-and-white check must reject it.

1The check passed

Average colour hid the variation between pixels.

False confidence exposed

2Repair the instrument

Check the colour range within each frame. Require the coloured control to fail and its desaturated twin to pass.

A test proves the fix is load-bearing

3Review the exact film

Inspect the delivered export and compare suspected defects with source footage.

A revised cut needs a new review

First prove the check can detect the problem. Then judge the result.

Decision 1

Prove the checks can fail before paying to scale

Under the render-validation plan of 2 September 2026, each check had to reject a deliberately broken input before paid testing began, on a ladder from 10 to 20 minutes and then two 60-minute runs. The receipt for that free demonstration is tied to a fingerprint of the checking code, so a missing receipt, a failed demonstration or a changed check blocks the paid step.

The black-and-white experiment shows why. An ungraded control passed because its average colour was near neutral, which hid the variation between pixels. The corrected check also measures the colour range within each frame, with paired controls: the coloured fixture must fail and its desaturated twin must pass. Disabling the range comparison makes the regression test fail.

The cost is calibration before any spend on scale. In exchange, a stale green result cannot pass the gate.

SourceCheck calibration + mutation test

The spend gate that blocks paid runs on a failed or stale calibration, the black-and-white false positive, and the test run that proves the fix.

Read the source

Decision 2

Accept the exact film against a written rubric

I separated controlled pipeline checks from judgments about real edits, and defined two human gates: are the intended film-wide decisions visible, and would I publish this film? The rubric samples three random parts plus the opening and closing minute, and names the exact export; a revised cut needs its own review.

A run on NASA footage on 10 October showed the limits of automated judgment. The post-export checks passed, while looking at the frames found oversized graphics covering speakers and slides that the AI reviewer had called appropriately sized. Comparing with the source worked the other way: suspected freezes were static slides already in the original.

This costs review time and adds human judgment. Automated review locates issues; frame inspection, source comparison and an explicit acceptance decide where its evidence runs out.

SourceAcceptance rubric + human review

The rubric’s gating rules, the recorded verdict for the reviewed export, and the caption gap a later NASA-footage run exposed.

Read the source

Results

A 9 October receipt records 29 of 29 checks proving they catch a planted fault, tied to the version of the checking code.

The black-and-white fix holds: the coloured control now fails on within-frame colour range (29.75, against 0.00 for the true black-and-white render) even though its average colour passed.

The export I reviewed on 8 September passed both human gates. The revised cut that followed needed its own viewing under the same rule.

From the mutation run

Test Suites: 1 failed, 1 totalTests:       1 failed, 37 skipped, 4 passed, 42 totalSnapshots:   0 totalTime:        6.695 s, estimated 11 s