# Agent Evaluation and Testing Harness

> A golden-set evaluation harness for a Claude agent: representative test cases run in bulk through the Message Batches API, graded by an LLM judge forced into deterministic scoring via structured outputs (never a prefill, which 400s on current models), with tool-call correctness tracked as a separate metric from output quality. New prompt or model versions must match or beat the versioned baseline before shipping - a regression gate, not a vibe check.

**Sub-marker:** CCD-F.5
**Domains:** CCDVF-D4 · Evaluation, Testing & Debugging, CCDVF-D1 · Agents & Workflows, CCDVF-D2 · Applications & Integration
**Exam weight:** CCD-F D4 - Evaluation, Testing & Debugging (2.6%, the lightest domain)
**Build time:** 20 minutes
**Source:** 🟡 Developer-cert production pattern
**Canonical:** https://claudearchitectcertification.com/scenarios/agent-evaluation-and-testing-harness
**Last reviewed:** 2026-07-12

## In plain English

Once an agent is in production, 'it seemed fine when I tried it' stops being a testing strategy. This scenario is the harness that replaces manual spot-checking: a versioned set of representative inputs with known-good expected behavior, run in bulk, graded automatically, and compared against a baseline before any prompt or model change ships. It exists because the two most common ways teams get burned are shipping a prompt tweak that silently breaks an edge case, and grading agents only on how the final text reads while ignoring whether they actually called the right tools in the right order.

## Exam impact

Evaluation, Testing & Debugging (D4) is the lightest domain at 2.6%, but it's still a scored domain, and it's the one most developers skip in practice. D1 (Agents & Workflows) covers what's being evaluated - the agentic loop and its tool-call sequence. D2 (Applications & Integration) covers running the harness at scale via Batches.

## The problem

### What the customer needs
- Know before shipping whether a prompt or model change made things better or worse.
- Grade what the agent actually did (tool calls, sequence), not just how the final text reads.
- Run evaluations cheaply at scale, not one request at a time against a live endpoint.

### Why naive approaches fail
- Grading only the final text lets an agent that skipped identity verification still score well because the refund text 'looked right'.
- Using a weaker or cheaper model as the judge than the system under test produces unreliable, inconsistent scores.
- Re-running evals ad hoc with no versioned golden set gives you no regression signal, every run is a fresh, incomparable snapshot.

### Definition of done
- Golden set versioned alongside the prompt/model it was built against
- Tool-call correctness scored separately from output-text quality
- Judge forced into structured, parseable output via output_config.format
- New versions gated on matching or beating the baseline pass rate before deploy

## Concepts in play

- 🟢 **Tool evaluation & testing** (`evaluation`), Golden set + grading pipeline
- 🟡 **Debugging LLM failures** (`debugging-llm-system-failures`), Replay via full content blocks
- 🟢 **Structured outputs** (`structured-outputs`), Deterministic judge scoring
- 🟡 **Batch API** (`batch-api`), Bulk eval execution

## Components

### Golden Set, versioned inputs + expected behavior

A repo-tracked set of representative test cases, each with an input and the expected stop_reason/tool sequence/output shape, versioned alongside the prompt it was built against.

**Configuration:** golden_set_v3.jsonl - one JSON object per line: {id, input, expected_tools, expected_shape}.
**Concept:** `evaluation`

### Batch Runner, Message Batches API

Submits the full golden set as a single batch job rather than sequential live requests, at half the per-token cost and without holding a connection open per case.

**Configuration:** client.messages.batches.create(requests=[{custom_id, params} for each golden-set case]).
**Concept:** `batch-api`

### LLM Judge, structured, deterministic scoring

A model at least as capable as the system under test, forced into a fixed JSON schema via output_config.format so scores are parseable and comparable across runs.

**Configuration:** output_config={"format": {"type": "json_schema", "schema": JUDGE_SCHEMA}} - never a prefill (400s on current models).
**Concept:** `structured-outputs`

### Regression Gate, baseline comparison

Compares the new run's pass rate against the versioned baseline and blocks deployment if it regresses, turning eval results into a CI-style check rather than a one-off report.

**Configuration:** if new_pass_rate < baseline_pass_rate - TOLERANCE: fail the deploy step.
**Concept:** `evaluation`

## Build steps

### 1. Build a versioned golden set

Each case pairs an input with the expected tool sequence and output shape. Version it in the repo alongside the prompt it was authored against.

**Python:**

```python
golden_set = [
    {
        "id": "refund-happy-path",
        "input": "I want a refund for order 42, it arrived damaged.",
        "expected_tools": ["verify_customer", "lookup_order", "process_refund"],
        "expected_stop_reason": "end_turn",
    },
    {
        "id": "refund-over-cap",
        "input": "I need a $900 refund on order 55.",
        "expected_tools": ["verify_customer", "lookup_order", "escalate_to_human"],
        "expected_stop_reason": "end_turn",
    },
]
```

**TypeScript:**

```typescript
interface GoldenCase {
  id: string;
  input: string;
  expectedTools: string[];
  expectedStopReason: string;
}

const goldenSet: GoldenCase[] = [
  {
    id: "refund-happy-path",
    input: "I want a refund for order 42, it arrived damaged.",
    expectedTools: ["verify_customer", "lookup_order", "process_refund"],
    expectedStopReason: "end_turn",
  },
  {
    id: "refund-over-cap",
    input: "I need a $900 refund on order 55.",
    expectedTools: ["verify_customer", "lookup_order", "escalate_to_human"],
    expectedStopReason: "end_turn",
  },
];
```

Concept: `evaluation`

### 2. Run the golden set through Message Batches

Submit every case as one batch job instead of sequential live requests - half the cost, and it doesn't hold connections open per case.

**Python:**

```python
def submit_eval_batch(golden_set, model="claude-sonnet-5"):
    return client.messages.batches.create(requests=[
        {"custom_id": case["id"], "params": {
            "model": model, "max_tokens": 4096, "tools": tools,
            "messages": [{"role": "user", "content": case["input"]}],
        }} for case in golden_set
    ])
```

**TypeScript:**

```typescript
async function submitEvalBatch(goldenSet: GoldenCase[], model = "claude-sonnet-5") {
  return client.messages.batches.create({
    requests: goldenSet.map((c) => ({
      custom_id: c.id,
      params: { model, max_tokens: 4096, tools, messages: [{ role: "user", content: c.input }] },
    })),
  });
}
```

Concept: `batch-api`

### 3. Score tool-call correctness separately from text quality

Extract the actual tool call sequence from the response and compare it to the expected sequence directly - exact-match grading, not judge opinion, for structured behavior.

**Python:**

```python
def score_tool_sequence(response, expected_tools: list[str]) -> bool:
    actual = [b.name for b in response.content if b.type == "tool_use"]
    return actual == expected_tools
```

**TypeScript:**

```typescript
function scoreToolSequence(response: Anthropic.Message, expectedTools: string[]): boolean {
  const actual = response.content
    .filter((b): b is Anthropic.ToolUseBlock => b.type === "tool_use")
    .map((b) => b.name);
  return JSON.stringify(actual) === JSON.stringify(expectedTools);
}
```

Concept: `evaluation`

### 4. Grade open-ended output with a structured LLM judge

Force the judge into a JSON schema so scores are parseable and comparable. Use a model at least as capable as the system under test, and never a prefill.

**Python:**

```python
JUDGE_SCHEMA = {
    "type": "object",
    "properties": {
        "correct": {"type": "boolean"},
        "reasoning": {"type": "string"},
    },
    "required": ["correct", "reasoning"],
    "additionalProperties": False,
}

def judge_output(case_input: str, agent_output: str, expected_behavior: str):
    return client.messages.parse(
        model="claude-opus-4-8",
        max_tokens=1024,
        output_config={"format": {"type": "json_schema", "schema": JUDGE_SCHEMA}},
        messages=[{"role": "user", "content":
            f"Input: {case_input}\nExpected: {expected_behavior}\nActual: {agent_output}\n"
            "Does the actual output satisfy the expected behavior?"}],
    ).parsed_output
```

**TypeScript:**

```typescript
const judgeSchema = {
  type: "object",
  properties: { correct: { type: "boolean" }, reasoning: { type: "string" } },
  required: ["correct", "reasoning"],
  additionalProperties: false,
} as const;

async function judgeOutput(caseInput: string, agentOutput: string, expectedBehavior: string) {
  const resp = await client.messages.parse({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    output_config: { format: { type: "json_schema", schema: judgeSchema } },
    messages: [{
      role: "user",
      content: `Input: ${caseInput}\nExpected: ${expectedBehavior}\nActual: ${agentOutput}\n` +
        "Does the actual output satisfy the expected behavior?",
    }],
  });
  return resp.parsed_output;
}
```

Concept: `structured-outputs`

### 5. Gate deploys on the versioned baseline

Compare the new run's pass rate to the checked-in baseline. A regression beyond tolerance blocks the deploy, the same way a failing unit test would.

**Python:**

```python
def check_regression_gate(new_pass_rate: float, baseline_pass_rate: float, tolerance=0.02):
    if new_pass_rate < baseline_pass_rate - tolerance:
        raise SystemExit(
            f"Regression: {new_pass_rate:.2%} vs baseline {baseline_pass_rate:.2%}"
        )
    print(f"OK: {new_pass_rate:.2%} (baseline {baseline_pass_rate:.2%})")
```

**TypeScript:**

```typescript
function checkRegressionGate(newPassRate: number, baselinePassRate: number, tolerance = 0.02) {
  if (newPassRate < baselinePassRate - tolerance) {
    throw new Error(`Regression: ${newPassRate} vs baseline ${baselinePassRate}`);
  }
  console.log(`OK: ${newPassRate} (baseline ${baselinePassRate})`);
}
```

Concept: `evaluation`

### 6. Replay failures from full content blocks, not just final text

When a case fails, inspect the entire content array (tool_use, tool_result, stop_reason per turn) to see exactly where the agent diverged, not just the final answer.

**Python:**

```python
def debug_failure(case_id: str, response):
    print(f"--- {case_id} ---")
    print(f"stop_reason: {response.stop_reason}")
    for block in response.content:
        if block.type == "tool_use":
            print(f"  tool_use: {block.name}({block.input})")
        elif block.type == "text":
            print(f"  text: {block.text[:200]}")
```

**TypeScript:**

```typescript
function debugFailure(caseId: string, response: Anthropic.Message) {
  console.log(`--- ${caseId} ---`);
  console.log(`stop_reason: ${response.stop_reason}`);
  for (const block of response.content) {
    if (block.type === "tool_use") console.log(`  tool_use: ${block.name}(${JSON.stringify(block.input)})`);
    else if (block.type === "text") console.log(`  text: ${block.text.slice(0, 200)}`);
  }
}
```

Concept: `debugging-llm-system-failures`

## Decision matrix

| Decision | Right answer | Wrong answer | Why |
|---|---|---|---|
| Grading tool-call sequences | exact-match against the expected sequence | LLM judge opinion on whether it 'seems right' | Structured behavior (which tools, what order) has a ground truth; exact-match is cheaper and more reliable than judge opinion for it. |
| Grading open-ended text quality | LLM judge, forced into structured output | exact string match against a reference answer | Natural-language quality has no single correct string, but scores must still be parseable and comparable across runs. |
| Judge model tier | at least as capable as the system under test | the cheapest available model, to save eval cost | A weaker judge produces unreliable grading, defeating the purpose of the harness. |
| Running large eval sets | Message Batches API | sequential live requests in a loop | Batches cut cost 50% and don't hold connections open per case, and eval runs are inherently latency-insensitive. |

## Failure modes

| Anti-pattern | Failure | Fix |
|---|---|---|
| CCDF-15 · Text-only grading | The judge scores only the final response text, missing that the agent skipped verify_customer. | Score tool-call sequence separately (exact-match) from output-text quality (LLM judge). |
| CCDF-16 · Weak judge model | Haiku 4.5 is used to judge output from a Sonnet-5-powered agent, producing noisy scores. | Use a judge model at least as capable as the system under test. |
| CCDF-17 · Unversioned golden set | The eval script's test cases live only in someone's scratch notebook, changing between runs. | Version the golden set in the repo alongside the prompt/model it was built against. |
| CCDF-18 · Prefill-based judge scoring | Judge prompt ends with an assistant-turn prefill like '{"correct": ' to force JSON shape. | Assistant-turn prefills return a 400 on current models. Use output_config.format (structured outputs) instead. |

## Implementation checklist

- [ ] Golden set versioned in the repo, paired with a prompt/model version (`evaluation`)
- [ ] Eval runs executed via the Message Batches API (`batch-api`)
- [ ] Tool-call correctness scored separately from output text quality
- [ ] Judge forced into structured output, never a prefill (`structured-outputs`)
- [ ] Regression gate compares new pass rate against the checked-in baseline
- [ ] Failures debuggable from full content blocks, not just final text (`debugging-llm-system-failures`)
- [ ] Cost and latency tracked alongside accuracy in the eval report

## Cost &amp; latency

- **Eval run cost (100-case golden set, Batches):** ~$0.30-$0.60, 50% Batches discount on Sonnet 5 traffic for 100 short-to-medium cases with tool use.
- **Judge scoring pass:** ~$0.02-$0.05 per case, A separate Opus 4.8 (or Sonnet 5) call per case for open-ended text grading; skip it for pure tool-sequence cases.

## Domain weights

- **CCDVF-D4 · Evaluation, Testing & Debugging (2.6%):** LLM Judge + Regression Gate
- **CCDVF-D1 · Agents & Workflows (14.7%):** What's being evaluated: the agentic loop
- **CCDVF-D2 · Applications & Integration (33.1%):** Batch Runner integration

## Practice questions

### Q1. Your eval harness scores an agent 95% pass rate, but production shows it frequently skips identity verification before processing refunds. What's the gap?

The harness is likely grading only final output text quality. Add a separate, exact-match check on the tool-call sequence (verify_customer before process_refund) - structured behavior needs structured grading, not judge opinion on the text.

### Q2. You use Haiku 4.5 to judge outputs from a Sonnet-5-powered agent to save on eval cost. Scores are inconsistent between runs. Why?

The judge model must be at least as capable as the system under test. A weaker judge produces unreliable grading, which defeats the purpose of a regression gate.

### Q3. Your judge prompt ends with {"role": "assistant", "content": '{"correct": '} to force JSON output. It fails on the current model. Why, and what's the fix?

Assistant-turn prefills return a 400 on Opus 4.6+/Sonnet 5. Use output_config.format with a json_schema (structured outputs) instead - it constrains the response shape without relying on a prefill.

## FAQ

### Q1. How big should a golden set be?

Start with 20-50 cases covering the main happy paths and known edge cases (policy boundaries, ambiguous intent, tool failures). Grow it every time production surfaces a new failure mode - each incident becomes a new golden-set case so it can never regress silently again.

### Q2. Should every case use an LLM judge?

No. Use exact-match grading for anything with a ground truth (tool sequence, structured output fields, stop_reason). Reserve the LLM judge for genuinely open-ended text quality, where there's no single correct string.

### Q3. How often should the harness run?

On every prompt or model change before it ships, as a CI-style gate, plus on a schedule (e.g. nightly) against production traffic samples to catch silent drift that wasn't triggered by a code change.

### Q4. What tolerance should the regression gate use?

Start conservative (e.g. 2 percentage points) and tune based on how noisy your judge scoring is. A tolerance of 0% will flag noise as regressions; too loose a tolerance lets real regressions through.

## Production readiness

- [ ] Golden set covers known production edge cases, not just happy paths
- [ ] Judge schema and prompt versioned alongside the golden set
- [ ] Regression gate wired into the deploy pipeline, not run manually
- [ ] Tool-call sequence checks cover every state-changing tool
- [ ] Failure replay tooling surfaces full content blocks, not truncated text

---

**Source:** https://claudearchitectcertification.com/scenarios/agent-evaluation-and-testing-harness
**Last reviewed:** 2026-07-12

**Evidence tiers**, 🟢 official Anthropic doc · 🟡 partial doc / inferred · 🟠 community-derived · 🔴 disputed.