"It Feels Smarter" Isn't Engineering: How We Evaluate LLM Applications

"Since we switched to the new model, the answers feel smarter." — that sentence is banned as a conclusion on our team. A feeling can't ship, and a feeling can't trigger a rollback either. After more than a year of building LLM applications and paying the model bills ourselves, the lesson is: an LLM application without an evaluation test set is software without tests — you don't know how good it is now, and you certainly won't know whether the next change made it better or worse. This article covers the evaluation method we actually use. You don't need an academic-grade framework; a spreadsheet is enough to start.

Why "trying it out" isn't evaluation

LLM output is probabilistic: the same prompt is good today, mediocre tomorrow. The three or five cases you try casually are too small a sample, and you tend to pick questions you already know well — the conclusion is almost guaranteed to be biased. Worse is the chain reaction of changes: you tweak the prompt to fix one case and silently break ten others — without a test set, you'll never notice. It's exactly the situation of writing code without tests and hand-clicking through every change, and the fix is the same: make verification automated and repeatable.

Step one: build your test set

The test set doesn't need to be big — representative beats large. Our approach:

1. Sample from real inputs. Pull real cases from production (de-identified), stratified by scenario: common questions as the bulk, edge cases, known landmine questions that have failed before, and questions that shouldn't be answered (out of scope, attempts to induce violations). A few dozen questions is enough to start; one or two hundred is plenty.
2. Define "what a good answer looks like" for each question. Not a gold answer for verbatim comparison (an LLM phrases things differently every time), but criteria: which points must be included, what must not appear (fabricated numbers, claims beyond the knowledge base), format and tone requirements.
3. Keep accumulating landmine questions. Every production incident sends its case into the test set — this is the most valuable source of questions, turning every crash into a permanent regression defense.

Step two: scoring — blind human review and LLM-as-judge, together

Blind human review is the most trustworthy baseline: strip the labels from two versions' outputs (old prompt vs. new prompt, model A vs. model B), randomize the order, and have reviewers pick the better one. The blinding is the crux — without hiding the labels, people unconsciously favor the version they just tuned. We've committed that sin ourselves.

LLM-as-judge handles scale: write the criteria into a judging prompt and let another model score every question. It lets you run full regressions on every change, but it has known biases — preferring longer answers, preferring particular styles, favoring models from its own family — so our discipline is: LLM judges run the daily regressions; blind human review makes the major calls (switching models, major prompt rewrites), with periodic spot-checks to see whether the two scoring methods have drifted apart.

Also: whatever can be verified programmatically shouldn't be verified by a model. Is the output valid JSON, does it contain banned words, is the length within range — check these with rules: fast, exact, and free.

Step three: regression gating — run before and after every change

The whole process collapses into one sentence: before any change ships (prompt, model version, retrieval strategy, parameters), run the test set — release only if the score holds or rises. A vendor releasing a new model version, or you switching providers to save cost, goes through the same pipeline. This is exactly how we manage "LLM segment selection" quality in our own video-clipping engine: we've accumulated a set of annotated cases for "where are this video's highlights," and every adjustment to the selection prompt triggers a regression — otherwise "it feels like it picks well" turns, three months later, into "customers asking why the highlights are all wrong."

"It feels smarter" is the language of small talk, not engineering. The language of engineering is: out of a hundred questions, the last version got 74 right; this version gets 81.

Three common pitfalls

• Test-set leakage into the prompt. Tuning the prompt against the test questions, then grading with the same questions — the score will look great and mean nothing. Keep tuning cases and acceptance cases separate.
• Testing only accuracy, never refusal. "Does it properly decline what it shouldn't answer" matters as much as "does it correctly answer what it should" — especially in compliance-sensitive scenarios. This is the same defensive line as engineering guardrails against hallucination.
• Too many metrics at once. A score each for correctness, completeness, tone, and format — and in the end nobody knows which one to look at. Start with a single primary metric (this question: pass/fail) and subdivide only when needed.

The version you can start today

Open a spreadsheet: column one is the input, column two the criteria, and each subsequent column is one version's output plus pass/fail. Thirty questions, twenty minutes of human review per change — this most primitive version already beats the majority of LLM applications out there shipping on vibes. Once question count and change frequency grow, automate it. If your team is about to push an LLM application into production and wants to build the evaluation layer right the first time, take a look at our AI adoption services.