The problem
What the customer needs
- Simple, high-volume requests answered cheaply and fast.
- Hard, high-stakes requests get the intelligence they actually need.
- A graceful fallback when the highest tier is rate-limited, instead of a hard failure.
Why naive approaches fail
- Routing every request to Opus 4.8 'to be safe' multiplies cost 5x with no measurable quality gain on the 80% of traffic that's routine.
- Routing every request to Haiku for cost savings degrades quality on genuinely complex requests that need more reasoning depth.
- No fallback tier on a 429/529 from the top model fails the whole request instead of degrading gracefully.
- Routing decision made in < 200ms via a cheap classification pass
- Escalation criteria documented and measured, not guessed
- Blended per-request cost tracked and trending down as routing improves
- Rate-limit on any one tier degrades to the next tier, never a hard failure
The system
What each part does
4 components, each owns a concept. Click any card to drill into the underlying primitive.
Router
Haiku 4.5 classification pass
A single cheap Haiku 4.5 call (or a lightweight heuristic) classifies request complexity/intent before the real request is dispatched to the right tier.
Configuration
router_resp = client.messages.create(model="claude-haiku-4-5", max_tokens=16, output_config={"effort":"low"}, ...). Output is a tier label, not the final answer.
Workhorse Tier
Sonnet 5 - default
The default destination for the large majority of production requests: near-Opus quality on coding/agentic work at a fraction of the cost.
Configuration
model="claude-sonnet-5", output_config={"effort":"high"} (the default; tune down for routine work).
Escalation Tier
Opus 4.8 - explicit criteria
Reserved for requests that meet a documented bar: long-horizon agentic work, high-stakes correctness requirements, or tasks that measurably fail on Sonnet 5 in the eval harness.
Configuration
model="claude-opus-4-8", output_config={"effort":"high"} or "xhigh" for the hardest coding/agentic subset.
Fallback Handler
graceful tier step-down
On a 429/529 from the escalation tier, retries the request one tier down rather than failing outright.
Configuration
except RateLimitError: retry with model=WORKHORSE_MODEL instead of the escalation tier.
Data flow
6 steps to production
Classify the request with a cheap Haiku pass
The router call itself must be cheap. Keep max_tokens tiny, effort low, and ask only for a tier label, not an explanation.
def classify_tier(request_text: str) -> str:
resp = client.messages.create(
model="claude-haiku-4-5",
max_tokens=16,
output_config={"effort": "low"},
system="Classify the request as one of: simple, standard, complex. Reply with one word.",
messages=[{"role": "user", "content": request_text}],
)
return resp.content[0].text.strip().lower()Route to a tier and set per-tier effort
Map the classification to a model + effort pair. Sonnet 5 is the default for 'standard'; escalate only 'complex' to Opus.
TIER_MAP = {
"simple": ("claude-haiku-4-5", "low"),
"standard": ("claude-sonnet-5", "high"),
"complex": ("claude-opus-4-8", "high"),
}
def route_request(tier: str):
model, effort = TIER_MAP.get(tier, TIER_MAP["standard"])
return model, effortDispatch on the chosen tier
The router's decision feeds a normal Messages API call - the routing logic and the actual request are two separate calls, not one combined prompt.
def dispatch(request_text: str, system_prompt: str):
tier = classify_tier(request_text)
model, effort = route_request(tier)
return client.messages.create(
model=model,
max_tokens=4096,
output_config={"effort": effort},
system=system_prompt,
messages=[{"role": "user", "content": request_text}],
)Fall back a tier on rate limit or overload
Catch typed rate-limit/overload exceptions on the escalation tier and step down one tier rather than surfacing a hard failure to the user.
def dispatch_with_fallback(request_text: str, system_prompt: str):
tier = classify_tier(request_text)
model, effort = route_request(tier)
try:
return client.messages.create(model=model, max_tokens=4096,
output_config={"effort": effort}, system=system_prompt,
messages=[{"role": "user", "content": request_text}])
except (anthropic.RateLimitError, anthropic.APIStatusError) as e:
if model == "claude-opus-4-8":
return client.messages.create(model="claude-sonnet-5", max_tokens=4096,
output_config={"effort": "high"}, system=system_prompt,
messages=[{"role": "user", "content": request_text}])
raiseMove non-interactive bulk work to the Batches API
Anything latency-insensitive (nightly re-scoring, bulk classification) should run through Message Batches for the 50% discount rather than live requests.
def submit_bulk_job(items: list[dict]):
return client.messages.batches.create(requests=[
{"custom_id": item["id"], "params": {
"model": "claude-sonnet-5", "max_tokens": 1024,
"messages": [{"role": "user", "content": item["text"]}],
}} for item in items
])Log routing decisions for a data-driven escalation policy
Every dispatched request logs the chosen tier, tokens, cost, and a downstream quality signal, so escalation criteria come from measured data, not intuition.
def log_routing_decision(request_id, tier, model, usage, quality_signal=None):
telemetry.insert({
"request_id": request_id, "tier": tier, "model": model,
"input_tokens": usage.input_tokens, "output_tokens": usage.output_tokens,
"quality_signal": quality_signal,
})The four decisions
| Decision | Right answer | Wrong answer | Why |
|---|---|---|---|
| Default model tier for production traffic | claude-sonnet-5 | claude-opus-4-8 for every request | Sonnet 5 is the current cost/quality default; Opus should be an explicit escalation, not the baseline. |
| Routing decision model | claude-haiku-4-5, tiny max_tokens, low effort | the same model as the actual request, run twice | The router only needs a tier label, not the answer - it should be the cheapest, fastest call in the pipeline. |
| Escalation criteria | measured failure patterns on the workhorse tier, tracked over time | gut feeling that a task 'seems hard' | Data-driven escalation avoids both over- and under-spending; an eval harness (Scenario CCD-F.5) is the source of that data. |
| Rate-limit handling on the top tier | step down to the workhorse tier and retry | surface a hard failure to the user | A slightly lower-quality answer beats no answer, and Opus rate limits should not take down the whole request path. |
Where it breaks
4 failure pairs. Each maps to an exam-style question - the naive move on the left, the disciplined fix on the right.
Every request, regardless of complexity, is sent to claude-opus-4-8 'to be safe'.
CCDF-11Route routine traffic through Sonnet 5 by default; reserve Opus for a documented, measured escalation criterion.
A 429 from the escalation tier propagates as a user-facing error.
CCDF-12Catch the typed rate-limit exception and retry one tier down before failing.
Code sets temperature=0 on Opus 4.8 expecting identical outputs every run.
Sampling parameters are removed on Opus 4.7+ and Sonnet 5 (400 error). Even where they existed, temperature=0 never guaranteed identical outputs - use prompting for consistency instead.
Cost model assumes Sonnet 5 token counts match Sonnet 4.6 for the same text.
CCDF-14Sonnet 5 uses a new tokenizer producing roughly 30% more tokens for the same input than Sonnet 4.6 - re-baseline with count_tokens rather than reusing an old multiplier.
Cost & latency
Router classification pass and any simple, high-volume traffic.
Default workhorse tier for the bulk of production requests.
Escalation tier only, gated by measured criteria, not the default path.
80% Haiku+Sonnet routine traffic, 15% Sonnet standard, 5% Opus escalation - illustrative, tune against your own traffic mix.
Ship checklist
Two passes. Build-time gates verify the code; run-time gates verify the system in production.
Build-time
- Router uses Haiku 4.5 with tiny max_tokens and low effort↗ model-selection-tradeoffs
- Sonnet 5 is the documented default tier for standard traffic
- Opus 4.8 escalation criteria are written down and measured, not assumed↗ accuracy-latency-cost-tradeoffs
- Rate-limit/overload on the top tier falls back one tier before failing
- Non-interactive bulk work runs through the Batches API↗ batch-api
- Per-tier cost and quality signals logged for ongoing router tuning
Run-time
- Router classification accuracy periodically spot-checked against human judgment
- Cost dashboard broken down by tier, trending as routing improves
- Fallback path tested under simulated rate-limit conditions
- Escalation criteria documented and reviewed, not tribal knowledge
- Batches API used for all latency-insensitive bulk workloads
Five exam-pattern questions
A team routes every production request through Opus 4.8 because 'it's the smartest model'. Cost is 5x a comparable Sonnet-5-default setup with no measurable quality difference on 80% of traffic. What's the fix?
Your escalation tier (Opus 4.8) returns a 429 during a traffic spike. The request currently fails outright. What should happen instead?
RateLimitError and retry the request one tier down (Sonnet 5) rather than surfacing a hard failure - a slightly different answer beats no answer, and the top tier's rate limit shouldn't take down the whole path.You're running a nightly job re-scoring 50,000 stored items with no user waiting on the result. Should this go through live Messages API requests?
Frequently asked
Isn't running a Haiku classification call before every request just adding cost and latency?
How do I decide the actual escalation criteria, not just 'when it feels hard'?
Should effort tuning happen before or after considering a bigger model?
output_config.effort on the current tier is cheaper than jumping a model tier, and often closes the gap - reserve the tier jump for cases where effort tuning alone doesn't get there.