On a film set, before the lead actor walks into frame, a stand-in walks the scene first. Same marks, same blocking, same light. The stand-in’s whole job is to let the crew rehearse the shot without risking the real performance. Nobody expects the stand-in to act. They just need to walk the path accurately enough that when the lead steps in, nothing surprises the camera.
That is the deal I made with a piece of code inside my AI Practice Companion, the RAG platform running on 346 of my own articles at ai-insight.directingbusiness.in. I call it the Shadow QA Fidelity Judge, and its design borrows from a paper Netflix published this year on exactly this problem: who evaluates the evaluator, once you have handed an LLM the job of judging your other LLMs’ work.
What Netflix actually built
The paper is “The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations” (Kong, Tan, Gupta et al., arXiv:2608.18300). Netflix generates the short explanations under each title on your home screen, “because you watched X,” at a volume no human review team could ever read. Their answer was not “trust the model.” It was to treat the judge itself as having a lifecycle with four phases.
Birth: a human-curated benchmark, labeled with a rationale for every failure, because that rationale matters enormously later. Training: the judge gets aligned to the benchmark through Reasoning-Aligned Rubric Tuning, where a meta-judge checks whether the judge reached the right verdict for the right reason, since a judge can land on a correct label through wrong reasoning and no accuracy metric will catch it. Deployment: the judge gates real output, and failed explanations get dropped rather than shown, because a bad explanation is a bigger trust hazard than a missing one. Monitoring: weekly, human-sampled, and the judge only has to stay within two standard deviations of how much human raters disagree with each other. Nothing about the rubric changes without a human signing off.
My build is a shadow of that, in both senses of the word. I borrowed the spirit: a judge earns trust before it earns influence, and a human-review gate sits between “the judge says so” and “the judge gets to decide anything.” What I built is closer to Phase Zero. Get the shadow judge observing, get its taxonomy right, and do not let it touch behavior until there is real evidence it agrees with a human.
The category that didn’t exist
My judge watches every turn of the Practice Companion’s Q&A, including the secondary 5-Fuse Business Triage mode I built for founders in financial distress, and logs a fixed vocabulary of outcomes to a Postgres table. No question text. No answer text. Just categories, shadow mode, non-gating.
The first thing it could say about any turn was whether the answer was grounded in my articles: pass or not_answerable_from_sources. A perfectly good axis for a RAG system. A completely useless axis for the question I actually cared about: when someone types something that sounds like financial trouble, does the app handle it well?
I found out the unglamorous way, by typing seven synthetic distress prompts into my own chat window and reading what came back. “Our biggest client just walked” got flatly redirected to the crisis triage. So did “things are tight right now, just tight, I’ll figure it out I guess,” a sentence with no business object in it at all. Meanwhile “money’s not the point, I’m just tight on time” got a warmer, more honest answer.
Here is the part that made the gap concrete: two of those seven turns, the flat redirect and the perfectly normal one, both logged pass. Read the CSV cold and you cannot tell which is which. The taxonomy was not wrong. It just could not see the thing I needed it to see.
Two fields, not one
The fix split the judgment into two independent axes: crisis_signal, what the message actually contained, and crisis_response, what the app did about it. A regex classifier now reads ownership language, negation, hypothetical framing, and explicit non-financial objects before deciding. The one behavioral bug the testing surfaced, that an ambiguous sentence got treated more harshly than an explicit financial one purely because it carried less information, got flipped: less certainty should soften the response, not sharpen it.
Nine synthetic prompts became the seed benchmark. After the fix shipped, five of nine matched exactly, including the one that mattered most. I want to be precise about what that is: nine mostly self-graded cases is not Netflix’s Birth and it is not Monitoring. It is evidence in the right direction, encoded as real unit tests rather than vibes.
The door nobody showed the stand-in
Then, verifying the fix live, one more prompt broke the pattern: “Cash flow’s been a bit inconsistent lately, but nothing I can’t handle yet.” By every rule the new classifier had just learned, this should have gotten a soft, substantive answer. Instead it walked straight into the full five-question crisis intake.
The classifier was never asked. The frontend has its own, older, cruder keyword trigger, “cash flow” among the phrases, that switches the whole conversation into triage mode before the careful logic ever runs. Two gates, guarding the same door, tuned by two different standards, built at two different times, and nobody had ever tested them together.
The judge’s own log made this legible in a way a screenshot could not: that turn recorded financial_distress with no response attached, because the response field never gets set once the frontend has already decided. Not a wrong answer. A blind spot the taxonomy could not even name yet.
I checked again
Six days after I first wrote this up, I got a written account of the fix: routing tightened, a hard/soft/normal split formalized, two benchmarks run clean. It read like the door had been closed. So I did the one thing this whole piece argues for, and did not take the account’s word for it. I typed the exact sentence that broke things the first time.
Straight into the five-question intake again. I republished, tried once more. Same result.
Then the isolating prompts told the real story. “Our cash position has been a little shaky but we’re managing” got exactly the right answer, triage offered only as an option. “I can’t make payroll this month and two vendors are threatening to cut us off” went straight to intake, correctly. The classifier handles the mild case and the severe case every time, except when the sentence contains the two words “cash flow,” at which point an older keyword check grabs the wheel.
That is the whole bug, precisely stated: a hardcoded string match sitting in front of logic that already does this properly. Not hard to fix. Just not yet fixed, whatever the write-up said.
What a stand-in is actually for
One real bug found and fixed and verified live, one new bug found while verifying the fix, and a taxonomy that had to grow because reality refused to fit it. The lifecycle Netflix describes runs from the rationale on the first labeled example to the sign-off before a rubric ever changes.
A stand-in who has walked one entrance flawlessly a hundred times has told you nothing about the entrance nobody thought to show them. The fix is not a smarter judge. It is more doors, tested on purpose, by more than one person, before you believe you are finished.
Mine still has at least one more to walk. I know exactly which one, and exactly what it logged when it missed it. Which is more than I could have said about the version of this system that shipped last week.
Key Takeaways
- An evaluator without a lifecycle is a second opinion that agrees with the first for the same reasons the first was wrong.
- A pass/fail taxonomy can be correct and still blind to the failure you care about. Split what the user said from what the app did.
- Two gates tuned by two standards will fail exactly where they overlap. Test them together.
- Do not take the write-up’s word for it. Reproduce the break, dated, twice.
(A version of this piece first appeared on Medium: https://medium.com/@LakshmiNarayana_U/the-stand-in-never-rehearsed-the-back-door-2ade269b4aa2)
