The LLM as Validator: What Separates a Demo From a System You Can Trust

Ask anyone what large language models do and they will say: generate. Text, code, answers, summaries. Generation is the demo, the headline, the thing that makes people gasp the first time they see it. But after shipping several AI systems to production, we have come to a conclusion that sounds almost backwards: the most valuable LLM in a serious system is usually not the one generating — it is the one checking.

We call the pattern the LLM as validator, and once you see it, you find it everywhere real money and real decisions are involved.

The demo failure everyone has seen

Here is a real example from a quoting engine we built. A customer types "cooking oil, 1 liter." Semantic search — embeddings, vector database, all the modern machinery — retrieves the closest products by meaning. And near the top of the results: motor oil. And a deep fryer.

Nothing malfunctioned. Embeddings measure closeness of meaning, and in vector space, a deep fryer genuinely lives near cooking oil. Retrieval did its job. The problem is that "close in meaning" and "is the thing the customer asked for" are different questions — and the second one is the only one the business cares about.

The fix was not better embeddings. It was adding a second model whose only job is to look at the candidates and answer: is this actually the requested product? With a strict rule for category (motor oil is never cooking oil) and a flexible rule for size (if there is no 1-liter bottle, 900 ml is a valid answer; a fryer never is). That validator is the difference between a demo that impresses and a system a distributor can put in front of customers.

The same pattern, three very different systems

Once we noticed it, the validator turned out to be the load-bearing wall in three systems that share nothing else.

In the quoting engine, the validator confirms every match before it reaches the quotation — and when nothing passes, the system says "not found" instead of quietly substituting something similar. An honest gap costs a follow-up question; a confident wrong answer costs a return, a complaint, and trust.

In document analysis for insurance and legal teams, the validation is citations: every extracted field points to the exact passage of the original document where it appears. The professional does not have to trust the extraction — they verify it in one click. The generating model does the reading; the verification design does the convincing.

In the recruiting system, validation takes the form of explanations: every candidate's ranking comes with its reasons, citing the CV. A score without an argument is not acceptable in a decision that affects people — and, less obviously, it is also not improvable, because nobody can tell you where it went wrong.

Three systems, one principle: generation proposes, validation disposes.

Why this works: the asymmetry nobody prices in

The economics underneath are simple and almost always ignored. In business systems, a wrong answer usually costs far more than no answer. A missing product in a quote triggers a question; a wrong product in a quote triggers a return. An unextracted date sends a human to read one page; a wrong date can lose a case. The demo optimizes for answering everything; production optimizes for never answering wrong. Those are different objective functions, and they lead to different architectures.

The validator is also cheap insurance in exactly the place where LLMs are strongest. Asking a model to generate the right answer from scratch is the hard direction. Asking it to judge whether a specific candidate matches a specific request — with both in front of it — is a far easier task, and models are correspondingly much more reliable at it. You are spending a small model call to convert "probably right" into "checked."

How to add one to your system

The recipe is less glamorous than the principle, which is a good sign. Give the validator a narrow question with the evidence in front of it — never "is this good?", always "is this candidate the product requested, yes or no, and why?". Write the asymmetric rules explicitly: which mistakes are fatal (category), which are tolerable (size). Design the honest exit: what the system says when nothing passes, because "I did not find it" must be a first-class answer, not a failure state. And log every validation with its reason — those logs are where the system's future improvements come from.

None of this shows well in a demo. All of it is why a system still has users a year later.