cat >> /home/claude/harness-engineering.md << 'HARNESS3EOF'
---
---
PART III — THE EVAL HARNESS
You've built the agent harness. Now: how do you know it works? Not "did it look good in a demo" — *know*, with numbers, that it does its job and keeps doing it as models and code change. That's the eval harness, and skipping it is the difference between engineering and vibes.
Chapter 8: Why evaluating agents is fundamentally harder than evaluating models
First, calibrate on why this is genuinely hard — harder than the model evaluation you met in the fundamentals guide.
Evaluating a base model is clean: <cite index="53-1">give it a question, compare its answer to a gold reference, score it. Clean, deterministic, repeatable.</cite> Agents break all three of those properties, for four compounding reasons:
- **Errors compound.** <cite index="49-1">A weak plan, wrong tool, or bad early assumption doesn't stay contained. It cascades through every step that follows, so the visible failure is often far downstream of the actual mistake.</cite> The bug you see at step 12 was born at step 3.
- **Long, autonomous horizons.** <cite index="49-1">Agents take many steps without supervision, so failures hide deep in the trajectory rather than in the final answer.</cite>
- **Non-deterministic trajectories.** <cite index="49-1">The same input can produce a different path on every run depending on state, memory, and tool outputs, so a single passing test tells you very little.</cite> Even with temperature pinned, runs vary.
- **Failure attribution is hard.** <cite index="49-1">With retrievers, tools, planners, and sub-agents all in play, an end-to-end score tells you that something broke, not which component broke.</cite>
The consequence that reshapes everything: **the unit of an agent eval is not a single answer — it's a complete trajectory.** As the standard definition now goes, the unit is <cite index="52-1">a complete agent trajectory: the user message, the tool calls the planner chose, the tool outputs, the retries, and the final response</cite>. You're not grading a sentence; you're grading a *run*. And because runs are non-deterministic, one run is a sample of size one — you need many.
This is why an eval harness is infrastructure, not a script: it has to run realistic inputs through the whole agent, repeatedly, capture full trajectories, and score them along multiple dimensions. <cite index="54-1">An eval harness is the infrastructure that runs your LLM evaluations end to end.</cite>
---
Chapter 9: The three levels — end-to-end, trajectory, component
The foundational structure of agent evaluation: you evaluate at **three levels**, because each answers a different question and catches different failures.
**End-to-end: did the task succeed?** The outcome level — did the agent produce the right final result? This is what users care about, but alone it's nearly useless for *improving* the agent, because a pass can hide a filthy trajectory (right answer, reached by accessing forbidden data — Chapter 12) and a fail tells you nothing about *where* it broke.
**Trajectory-level: was the path efficient and sound?** <cite index="49-1">Was the path efficient and sound?</cite> Did the agent call the right tools in a reasonable order, without needless steps, retries, or detours? This is where you catch a right-answer-wrong-reasons pass and a plan that technically worked but burned 40 steps doing it. Trajectory metrics include tool-call correctness, plan quality, plan adherence, and step efficiency.
**Component-level: which piece broke?** <cite index="49-1">Which retriever, tool, or sub-agent broke?</cite> When something fails, you need to attribute it — was it the retrieval (wrong chunks), the planner (wrong strategy), a specific tool (bad output)? This is where your RAG-level metrics (recall@K, groundedness from the fundamentals guide) live, now as *components* of a larger agent eval.
The discipline: **measure all three, and use tracing to connect them.** <cite index="49-1">Tracing is the backbone: it shows where a metric failed, surfaces new failure modes you don't have metrics for yet, and — paired with periodic human review — keeps your evals calibrated as the agent drifts.</cite> The observability layer from Chapter 3 (the trajectory logs) is precisely what makes three-level evaluation possible. This is the concrete link between the two harnesses: **the agent harness must emit trajectories so the eval harness can grade them.** Build them together or the eval harness is blind.
A useful failure taxonomy to have in your head when reading trajectories — the common agent failure types: wrong tool selected, wrong arguments, wrong sequencing, tool hallucination (claiming a tool was used that wasn't), consistency failures (fails on re-runs), and reasoning failures (right answer, flawed steps). When an eval fails, you're classifying it into one of these, and the fix differs for each.
---
Chapter 10: Graders — deterministic, model-judge, human
*How* do you actually score a trajectory? Three grader types, and choosing correctly per-metric is the craft. This is the same three-tier structure serious harnesses (including the ones Anthropic describes) converge on:
**Code-based / deterministic graders (use wherever possible).** Plain code that returns an objective pass/fail: <cite index="47-1">tool call verification, schema validation, decision matching — deterministic, fast</cite>. The gold standard, because <cite index="43-1">they return an objective pass/fail the model cannot argue its way around</cite>. The rule to tattoo on your wall: <cite index="52-1">the scoring functions you can implement deterministically should be — schema conformance, tool-call recall and precision, latency budgets, token budgets, forbidden-tool checks, source-span substring validation: all of these are code, not LLM calls</cite>. For your legal engine, "does every citation point to a real retrieved chunk?" is a deterministic check — and a powerful one.
**Model-based / LLM-as-judge graders (for the genuinely subjective).** A second model scores the output against a rubric — <cite index="47-1">faithfulness assessment via LLM judge — flexible but introduces variance</cite>. Necessary for things you can't check mechanically: response coherence, tone, whether an answer is genuinely faithful to sources. But handle with care: an LLM judge <cite index="43-1">can be gamed or can collude with the actor</cite>, and it introduces its own randomness. Reserve it, per the standard guidance: <cite index="52-1">save LLM-judged scoring for genuinely subjective dimensions like response coherence and tone match</cite>.
**Human graders (the calibration gold standard).** <cite index="47-1">Audit sampling for calibration — gold standard, expensive.</cite> You can't human-grade everything, but you sample: humans periodically review a slice to check that your deterministic and LLM-judge graders are actually tracking truth. Humans are how you keep the automated graders honest.
The design principle: **deterministic where you can, LLM-judge where you must, human to calibrate both.** A common beginner mistake is reaching for an LLM judge for things that are actually deterministic (wasting money and adding variance to check something code could verify exactly). Push as much scoring as possible down to code.
---
Chapter 11: Golden tests, sampling, and drift detection
Now the operational mechanics — how the eval harness runs day to day, and the subtleties that separate a trusted eval suite from an abandoned one.
**Golden tests.** The core artifact: <cite index="52-1">5 to 10 golden test cases per agent capability. Each case carries an input, an expected trajectory (tools called, tools forbidden, max retries), an expected structured output, and a list of scoring functions.</cite> This is your agent-era version of the eval set you built for RAG — but richer, because it specifies the *expected trajectory*, not just the expected answer. Build these *with the domain experts* (the lawyers), including the nasty cases: questions with no answer in the corpus (correct behavior: refuse), forbidden actions the agent must *not* take, ambiguous inputs.
**Sampling — the non-determinism fix.** The subtlety that trips everyone: because runs vary, <cite index="52-1">a suite that fails on a single case after one run is not a regression; it is a sample of size one</cite>. The fix: <cite index="52-1">each case runs 3 times in CI. Reported score is the median. A flaky case that passes twice and fails once still passes the threshold.</cite> You're doing statistics, not unit testing — one run tells you almost nothing.
**Baselines and drift detection.** You <cite index="52-1">pin a baseline — a thresholds file pins acceptable scores per metric. A merged PR updates the baseline if scores improved; a failing PR is judged against the baseline.</cite> Then you watch for **drift** over time, and here's the judgment call that keeps the suite alive: <cite index="52-1">a drift run that drops 0.5% week-over-week is noise. A drop of 3% is a regression. Set the alert threshold high enough that the team trusts the alert. Teams turn off their eval suite when it is too noisy.</cite>
That last point is the most important operational lesson in this entire guide: **an eval harness that cries wolf gets ignored, and an ignored eval harness is worse than none** (it gives false confidence). The fix for noise is almost never lowering the quality bar — it's <cite index="52-1">increase the sample size, pin a baseline, and stop alerting on case-level noise</cite>. A trusted, slightly-less-sensitive alarm beats a precise alarm everyone mutes.
Why drift matters especially for you: model providers update models silently, and your Arabic-model options (Falcon, Jais) ship new versions monthly. Your eval harness is the regression alarm that tells you when a model update — or your own code change — quietly broke citation accuracy or Arabic legal fidelity. Without it, you find out when a lawyer does.
---
Chapter 12: The security gate — auditing the trajectory, not the answer
This chapter is where harness engineering and the security guide fuse, and it contains the single most important idea in agent evaluation that most teams miss.
**A correct answer can hide a security disaster.** Output-level evaluation is blind to a whole class of failure: <cite index="50-1">a harness can return a correct, benign answer while along the way accessing unauthorized resources, leaking private context to the wrong agent, or triggering irreversible side effects outside the intended scope. Evaluating only the final response misclassifies these runs as successful.</cite> For a legal engine, this is the nightmare: the agent gives a perfect answer to a question about matter X, but *reached it by reading matter Y* it had no right to touch. End-to-end eval scores that a success. It's a confidentiality breach.
The consequence, stated as a principle: **agent safety must be evaluated on the trajectory, not the response.** The research framing checks three properties jointly: <cite index="50-1">whether actions stay within the permission and information-flow boundaries the harness specifies (boundary compliance), whether the trajectory reaches the goal through valid intermediate steps (execution fidelity), and whether both survive realistic perturbations such as indirect prompt injection, ambiguous goals, and tool errors (system stability)</cite>.
**The security gate as a scoring primitive.** The cleanest way to encode this is a binary gate that dominates the score. In one benchmark's formulation, a task's score is `Security × Completion × Process` — where <cite index="55-1">Security is set to 0 if the run violates explicit permission or security constraints, such as unauthorized access, secret exposure, or forbidden actions; otherwise 1</cite>. Multiplying by a binary security term means **any security violation zeroes the entire score, no matter how good the answer was.** That's exactly the right incentive: a beautiful answer obtained by unauthorized access is worth zero.
For your legal engine, this translates into concrete deterministic checks in your eval harness: did any tool call in the trajectory access a matter outside the user's authorization? Did the agent ever attempt a forbidden action? Did it leak one client's context into another's? These are code-based graders (Chapter 10), they run on the trajectory logs (Chapter 9), and a single violation fails the case regardless of answer quality. This is also how you *test* your prompt-injection defenses (security guide): include adversarial cases — documents with hidden injection payloads — in your golden tests, and verify the security gate stays green. You're not hoping injection fails to work; you're *measuring* that it fails to work, every CI run.
HARNESS3EOF
echo "Harness Part 3 written"