- ✓UK AI Security Institute scored Mythos at 83.1% on CyberGym versus 81.5% for GPT-5.5, and Anthropic's May 18 disclosure shows Mythos generated 181 working Firefox exploits in a single automated run. The lesson is not which model wins a leaderboard; it is that temperature 0.1-0.2 + isolated attacker/defender contexts + a human patch-validation gate is the real architecture. Without that scaffold, GPT-5.5 will cheerfully fix a buffer overflow and ship a fresh logic bug (JDSupra, May 17).
- ✓AISI's 83.1% vs 81.5% is only meaningful if you understand why benchmark variance matters less than the validation architecture you wrap around the model. Evaluation-as-architecture is the lens this post reaches for.
- ✓A security audit is an agentic loop with the highest possible stakes: every iteration that ships an unreviewed patch is a regression risk. The loop shape, not the model, decides whether the workflow is safe.
- ✓Mythos's 500k window is the reason whole-repo audits work. Logic flaws cross file boundaries; the context architecture is what makes cross-module reasoning possible at all.
Quick answer
Mythos beat GPT-5.5 by 1.6 points on CyberGym (AISI, May 20) and surfaced 19 vulnerabilities with zero false positives in a Symfony core audit (May 19). The architecturally useful detail is not the leaderboard position; it is that source-code audits run at temperature 0.1-0.2, attacker and defender contexts must be isolated to prevent the model cheating, and a human patch-validation gate is mandatory because GPT-5.5 has shipped fresh logic bugs while fixing buffer overflows (JDSupra, May 17). Treat the benchmark as a feature flag, not a procurement decision.
Why benchmark leaderboards mislead in security
The popular story this week is that Mythos beat GPT-5.5 at hacking. Useful headline, true on the AISI scorecard, and almost entirely beside the point for any security team actually deploying these models.
Benchmark wins move procurement. They do not move the failure modes that get security teams fired. The failure mode that matters is the one JDSupra documented on May 17: an auto-patching agent fixes a buffer overflow and quietly introduces a logic bug nobody reviews because "the model said it was done". The 1.6-point gap on CyberGym has no opinion on that. The architecture around the model does.
What follows is the four-shift reading of the AISI and XBOW data - what actually changes in your workflow when you stop treating model selection as the whole answer.
Four shifts the AISI + XBOW data should produce
Shift 1. Model selection becomes a per-stage decision, not a procurement one.
The honest reading of AISI's column-by-column results is that no model wins every column. Mythos leads CyberGym overall (83.1% vs 81.5%) and SWE-bench Verified (93.9% vs 91.2%), and finished a multi-stage attack first. GPT-5.5 leads highest-difficulty success rate (71.4% vs 68.6%) and is roughly 2x faster on login-style tasks (XBOW, May 22). The architecturally honest workflow uses both: GPT-5.5 for recon and dead-end pruning, Mythos for deep source-code audit. Treating this as a single-vendor decision throws away free precision and free speed.
Shift 2. Temperature is a security control, not a creativity knob.
The-Decoder's May 20 writeup of practitioner Mythos settings is unusually specific: temperature 0.1 to 0.2 for source-code audits, 0.7+ reserved for creative red-teaming. The reason is deterministic vulnerability identification. At 0.7 the model produces speculative findings that look like bugs and waste reviewer time. At 0.1 the same model produces fewer, sharper findings and marks the rest as Requires Manual Review. The temperature setting is the difference between an audit report you can act on and a noise generator you have to triage.
Shift 3. Whole-repo context changes what counts as a finding.
Mythos's 500k token window is the reason the Symfony audit (May 19) found cross-module logic flaws that a file-at-a-time scanner would miss. Logic flaws do not stay politely in one file; race conditions cross async boundaries, authorisation bugs cross controller and middleware, and CORS misconfigurations cross the entire request lifecycle. The 19 vulnerabilities with zero false positives is the headline number. The architectural finding is that cross-module reasoning requires cross-module context, and the 500k window is what makes that mechanically possible. Without it, you get a stack of single-file warnings nobody can correlate.
Shift 4. The Validate step has to include credential exfiltration, not just correctness.
JDSupra's May 17 note on auto-patching GPT-5.5 introducing fresh logic bugs is the headline failure mode. The quieter one is in MindStudio's May 22 writeup: if you fail to isolate attacker and defender prompt chains, the model cheats by using leaked context from the defence strategy it just generated. Your eval is not measuring what you think it is measuring. The Validate step in any security agent loop has to include a human patch review (catches the buffer-overflow regression), context isolation between adversarial roles (catches the cheating), and a check that the agent did not write secrets to unintended paths. "Tests passed" is not a security signal.
The nuanced point
The Mythos vs GPT-5.5 framing flatters both vendors and helps neither security team. What the AISI and XBOW data actually argue is that the model is one component in a security-audit architecture, and the architecture is what determines whether you ship safer code or merely faster code.
That architecture, as practitioners are running it in May 2026, is recognisably a purple-team AI loop: reconnaissance and dead-end pruning on the faster model, deep audit and PoC generation on the more precise one, low-temperature settings throughout, isolated attacker/defender contexts to keep the evaluation honest, and a human patch-validation gate before any code lands. The leaderboard tells you which model to put in the audit slot this quarter. It tells you nothing about whether the slot exists in your workflow at all. Most teams asking "Mythos or GPT-5.5?" have not built the slot yet.
How this shows up on the exam
D1 (Agentic Architecture, 27%) reliably presents scenarios that look like model-selection questions and are actually loop-design questions. A security agent produces inconsistent patch quality; the distractors offer "use the higher-scoring model" or "increase context window size". The architecturally correct answer is almost always introduce a human review gate between Implement and merge or isolate adversarial contexts so the evaluation is not contaminated. The exam rewards candidates who reach for structural answers before tooling answers.
D3 (Operations, 20%) and D5 (Context Engineering, 15%) carry the rest of this material. D3 tests whether you treat benchmark scores as an input to architectural decisions rather than a substitute for them - questions about "the new model scores 2 points higher on a security benchmark, should you migrate?" almost always reward the answer that asks about regression risk, false-positive rate in your specific codebase, and review-gate coverage first. D5 shows up whenever a scenario describes cross-file or cross-module vulnerability discovery; the correct primitive is whole-repo context, and the trap answers offer per-file scanning at higher temperature for "creativity". Creativity is not what you want in a vulnerability report.
What part of your security workflow are you still letting the model close out unreviewed?
The honest answer for most teams: patch validation. The model fixes the bug, the test suite goes green, the PR gets merged. The buffer overflow is gone. The fresh logic bug is not. The benchmark gap between Mythos and GPT-5.5 is small enough to be noise; the gap between teams with a human patch-validation gate and teams without one is the gap that actually shows up in post-mortems.
4 direct routes into the scored material.
- conceptsAISI's 83.1% vs 81.5% is only meaningful if you understand why benchmark variance matters less than the validation architecture you wrap around the model. Evaluation-as-architecture is the lens this post reaches for.
- conceptsA security audit is an agentic loop with the highest possible stakes: every iteration that ships an unreviewed patch is a regression risk. The loop shape, not the model, decides whether the workflow is safe.
- conceptsMythos's 500k window is the reason whole-repo audits work. Logic flaws cross file boundaries; the context architecture is what makes cross-module reasoning possible at all.
- knowledgePurple-team AI (GPT-5.5 for recon, Mythos for audit, humans for patch validation) is an architecture decision before it is a tooling decision. This post is a worked example of that pattern.
Share this post.
One share is one less person stuck on the same question.