On this page
TLDR
Evaluation is how you measure tool-call correctness against ground-truth datasets. A full deep-dive guide is coming soon.
What you are deciding
A prompt change is an improvement only when it beats the current prompt on the same held-out cases, against criteria written beforehand. An average that rises can hide a regression on the cases that matter most.
Does this change improve the measured outcomes, or only the examples I happened to look at?
What it is
An evaluation (or eval) is a structured test harness that runs prompts and agents against a suite of test cases, scores each result, and measures improvement over iterations. Unlike manual spot-checking, evals are automated and repeatable: the same 50 cases run against Prompt-v1, Prompt-v2, Prompt-v3, with scores improving from 68% to 87%. Evals are the difference between hope (a prompt feels good) and proof (test cases show +13 points).
The eval workflow has four phases. Define test cases (30-100 input-expected pairs). Run agents against each case, capture response and tool calls. Score results with code-based or model-based grading. Analyze and iterate: which cases failed and why? Refine the prompt, re-run, measure delta. Success is when the score plateaus or new cases reveal new failure modes (regression detection).
Golden test suites are the foundation. Golden means the expected outputs are verified (human-reviewed, audited, canonical). 50 cases where only 10 are golden produces misleading scores. Build golden incrementally: week 1 = 10 golden cases; month 2 = 50; month 6 = 200. The exam tests whether you understand that evals without golden data are theater, not measurement.
Tool-calling evals are distinct from output evals. An output eval scores final text. A tool-calling eval scores behavior: did the agent call the right tools in the right order with the right arguments? For agents, tool-call evals matter more, the agent's job is orchestration, not prose. The exam drills this distinction repeatedly.
How each certification frames it
CCA-P: How CCA-P tests evaluation and iteration
CCARP-D4-O1Define evaluation metrics (accuracy, latency, cost, safety, security)CCARP-D4-O2Design evaluation datasets and test frameworks using mixed methodologiesCCARP-D4-O3Conduct A/B testing and iterative improvements
The Professional guide lists three linked tasks in Domain 4: define evaluation metrics (accuracy, latency, cost, safety, security), design evaluation datasets and test frameworks using mixed methodologies, and conduct A/B testing and iterative improvements (PDF page 5).
Anthropic's evaluation guidance asks for success criteria that are specific, measurable, achievable and relevant, and for evals that mirror the real task distribution, edge cases included. Expect items where several options sound careful and exactly one keeps the comparison controlled.
How it works
An eval harness is three components: test data, scoring function, and a loop. Test data is JSONL (one case per line): {input, expected_tools, expected_outcome}. The loop runs each case through the agent, parses tool calls, and feeds both to the scoring function. The function returns a numeric score (0-1) or pass/fail. Aggregate across all cases, run on Prompt-v1, refine, run on v2, measure delta.
Code-based scoring is deterministic: parse the agent's response, extract tool calls, compare against expected. expected == actual → pass. Works for tool-call evals, JSON schema validation, rule-based checks. 100% reproducible, no hallucination. Use for anything measurable.
Model-based scoring lets Claude grade subjective cases (tone, completeness, empathy). Pass the input, agent output, and rubric to Claude; Claude returns a structured score. Useful but not reproducible across model versions. Use sparingly, always verify a few examples manually.
Regression detection is the hidden value. Run evals monthly: month 1 = 85%, month 2 = 84% after a system prompt edit. Regression detected. Roll back or understand the trade-off. Without evals, the regression goes unnoticed for weeks. With evals, caught immediately.
Worked example: comparing two ticket-classification prompts
Authored teaching example, not a bank question.
A support team wants to replace its ticket-classification prompt (A) with a newer one (B). All figures below are invented for teaching and are not measurements.
| Slice | Cases | Prompt A correct | Prompt B correct |
|---|---|---|---|
| Billing | 100 | 88 | 94 |
| Shipping | 80 | 72 | 76 |
| Urgent safety | 20 | 19 | 15 |
| All cases | 200 | 179 | 185 |
The average said B was better; the slice said it was unsafe. Check afterwards that the criteria were written before the run, that only one variable changed, and that the held-out set was never used to tune either prompt.
How this decision goes wrong
Changing several variables at once
Looks right: The new version has a better prompt, a newer model and a different output format, so it should win.
Why it fails: When quality moves you cannot say which change caused it, so you cannot keep the good change and drop the bad one.
Instead: Change one variable per comparison, or run each change as its own labelled experiment on the same cases.
Tuning against the test set
Looks right: The failing test tickets show exactly what to fix, so the prompt is edited until they pass.
Why it fails: The set now guides the edits, so a high score on it shows memorized fixes, not generalization.
Instead: Keep a development set for editing and a held-out set for judging. Refresh the held-out set when it has leaked into edits.
Declaring victory from one example
Looks right: B handled the ticket that A got wrong, so B is better.
Why it fails: One case cannot show a trend, and model output varies. Anthropic's guidance favours more cases over fewer, higher-effort ones.
Instead: Compare on a representative set with the criteria agreed in advance, and look at slices as well as the average.
Where you'll see it
Golden test suite for refund agent
50 cases: 10 happy path, 10 edge cases, 20 escalations, 10 errors. Run, score, refine. After 3 iterations, score climbs from 80% to 95%. Failures route to the 5% that need human review. Eval-driven refinement beats manual spot-checking by 4x in time-to-production.
Regression detection in research agent
Daily eval on 100 cases. Tuesday baseline 82%. Wednesday's prompt refinement scores 79%. Regression caught in hours, not weeks. Rollback or adjust. Without evals, 50+ bad outputs would ship before users complained.
Show 2 more examples
Tool-call eval for CI/CD bot
30 PRs, expected tool sequence [Read, Grep, Bash, Report]. 28/30 pass. The 2 failures called Report twice or skipped Bash. Refine tool descriptions, re-run: 29/30. The eval prevents shipping a bot with incomplete analysis.
Continuous evaluation in shadow mode
Deploy improved agent in parallel (no user impact). Score 1000 real interactions. 94% pass. Fix the 6% (wrong tool, invalid customer_id), redeploy: 98%. Once at 95%+, ship to production. Evals are continuous, not one-time.
Code examples
import json
def score_agent(test_case: dict, agent_response: dict) -> dict:
"""Compare actual tool calls to expected."""
expected_tools = test_case["expected"]["tools"]
actual_tools = [b.name for b in agent_response.get("tool_calls", [])]
score = 0
if actual_tools == expected_tools:
score += 50 # Tool sequence correct
if test_case["expected"]["outcome"] == agent_response.get("outcome"):
score += 30 # Outcome matches
if test_case["expected"].get("reason") in agent_response.get("reason", ""):
score += 20 # Specific reason correct
return {"score": score / 100, "case_id": test_case["id"]}
def run_eval_suite(cases_path: str, agent_fn):
cases = [json.loads(line) for line in open(cases_path)]
scores = []
for case in cases:
response = agent_fn(case["input"])
result = score_agent(case, response)
scores.append(result["score"])
if result["score"] < 1.0:
print(f"FAIL case {case['id']}: {result}")
avg = sum(scores) / len(scores)
print(f"Avg: {avg:.2%} ({sum(1 for s in scores if s == 1.0)}/{len(scores)})")
return avg
run_eval_suite("refund_cases.jsonl", refund_agent_v2)Looks right, isn't
Each row pairs a plausible-looking pattern with the failure it actually creates. These are the shapes exam distractors are built from.
Run a few manual examples to verify the prompt works, then deploy.
Manual spot-checks miss edge cases and provide no proof.
A prompt working on 5 chosen examples might fail on 10% of real traffic. Evals run 50-100 diverse cases and catch corner cases.
Evals are only for NLP text quality; tool-using agents don't need them.
Tool-using agents benefit more from evals.
Tool calls are deterministic and crisp to measure: did it call the right tools in the right order with the right arguments?
If evals score 95%, the agent is production-ready.
Evals score only on test cases you designed.
Missing scenarios (international, mobile, peak load) don't show up. 95% on the suite is necessary but not sufficient. Pair with production monitoring.
Use model-based grading for all eval scoring; it's fast and objective.
Model-based grading fails on deterministic measurements.
Use it for subjective cases only (tone, completeness). For tool sequences and JSON schemas, code-based scoring is unambiguous and reproducible.
Generate test cases by running the agent and treating its output as expected.
That's circular validation.
You're measuring whether the agent repeats itself, not whether it's correct. Golden cases must be externally validated (human review, expert sign-off).
Side-by-side
| Type | Test Data | Scoring | Reproducibility | Best For |
|---|---|---|---|---|
| Code-based (tool calls) | Expected tool sequence | Exact match | 100% | Agent behavior, compliance |
| Code-based (output) | Expected JSON schema | Schema validation | 100% | Structured output |
| Model-based | Rubric + expected | Claude grades | Not reproducible | Subjective (tone, completeness) |
| Hybrid | Code checks + rubric | Code first, model fallback | Mostly reproducible | Complex cases |
| Production monitoring | Real interactions | Live scoring | Always current | Catching real-world regressions |
| Manual spot-check | Hand-picked examples | Subjective judgment | Not reproducible | Quick sanity check (not proof) |
Decision tree
Can the correct outcome be checked with code (exact match, schema validation)?
Do you have 50+ test cases that have been externally validated?
Is the metric tied to business impact (compliance, revenue, legal)?
Will you run evals only once or continuously?
Are there edge cases you haven't tested yet?
The exam trap
Declaring a prompt or tool change better from a rising average or from a single example. Compare the change with the current version on the same held-out cases, using criteria written beforehand, change one variable at a time, and look at slices as well as the average, because a higher average can hide a regression on the cases that matter most.
Question patterns
CCA-P practice questions
3 approved CCA-P questions for this topic. Tap an answer to check it, then read the rationale for every option.
A team building an eval suite for a document-classification agent hand-writes 20 test cases, all drawn from documents they personally found easy to classify. The suite passes at 100%, but production accuracy is far lower. What is the core flaw in the eval dataset design?
CCA-F practice questions
3 approved CCA-F questions for this topic. Tap an answer to check it, then read the rationale for every option.
Your code-review CI bot returns valid JSON with empty findings: [] even on PRs that clearly have issues. Why?
Frequently asked
Showing 13 of 13 questions
[A, B, C] OR [B, B, C] may both pass.Sources and next steps
- Claude Certified Architect - Professional Exam Guide v1.0 (July 2026), PDF page 5, section: Domains 3, 4 and 5, detailed objectives. Checked 2026-10-16.
- Create strong empirical evaluations (Claude API docs), section: Define your success criteria; Eval design principles. Checked 2026-10-16.
CCA-P exam guide: Domain 4, Evaluation, Testing & Optimization