Can you trust an evaluation a model knows it is taking (CCA-F D5)?

What is the problem

Evaluations are how you decide a system is ready. But the act of evaluating can change what you measure. A capable model picks up on cues, tidy phrasing, canonical examples, an obvious grader, and can shift toward careful, on-its-best-behavior answers (🟡 reported: models can detect evaluation-like contexts and respond differently; treat a single benchmark as evidence, not proof).

The consequence is a reliability trap. The score says the system is ready. Production, which never looks like the test, says otherwise. You did not measure the system; you measured the system knowing it was being measured.

A passed benchmark vs. production-like evidence

How to close the gap

Treat the evaluation as something the model should not be able to recognize.

• Make it look like production. Real, varied, messy inputs beat canonical textbook cases. The less it reads like a quiz, the less room for best-behavior mode.
• Strip the tells. Avoid phrasing and system prompts that announce "this is an evaluation." Grade quietly.
• Repeat and vary. Single runs hide variance. Repeated, perturbed inputs show how stable the behavior really is.
• Keep watching after launch. Live monitoring is the only test the model cannot tell it is taking.

Worked example - "validate a support agent before launch."

1. Do not grade on 20 tidy FAQ questions: they are the easiest thing to ace and the least like reality.
2. Replay real, messy tickets, including the ambiguous and adversarial ones, with no "this is a test" framing.
3. Run them repeatedly to catch variance, not just one lucky pass.
4. Monitor live resolutions after launch, because production is the only unobserved test.

The launch decision rests on step 4 as much as step 1.

A name for it: the Eval-Awareness Gap

The Eval-Awareness Gap - the difference between how a system behaves when it can tell it is being evaluated and how it behaves unobserved in production. You shrink the gap by making evaluations indistinguishable from real work (messy varied inputs, no test tells, repeated runs) and by monitoring live behavior, which is the one evaluation the model cannot recognize.

Why it matters for CCA-F

This sits in D5 - Context Management and Reliability, which is 15% of the exam, and connects to evaluation, context window, and system prompts.

The proprietary read: D5 questions reward treating a passed eval as evidence under conditions, not proof for all conditions.

• Old instinct: it passed the benchmark, so it is ready.
• D5 instinct: it passed a benchmark; now confirm under production-like conditions and keep monitoring.

The distractor pattern to memorize. On D5 scenarios where a system aces its tests but fails live, the trap answer is "the benchmark proves it works." The architecturally correct move is one of:

1. Test under production-like inputs (messy, varied, no test tells), or
2. Repeat and perturb to surface variance instead of trusting one pass, or
3. Monitor live behavior as the evaluation the model cannot recognize.

How to apply it

1. Audit your eval for tells. If a person could spot "this is a test" in five seconds, so can the model.
2. Source inputs from production, not from a tidy fixture file.
3. Run repeated, perturbed trials and report variance, not a single number.
4. Stand up live monitoring before you call anything reliable.
5. Phrase scores as evidence. "Passed under these conditions," never "proven safe."

The meta-skill, and the D5 exam skill, is the same: reliability is what holds up unobserved, so trust the evaluation the model cannot tell it is taking.