What this system is, and its 5 parts
Think of this as the way you turn a messy email or a 200-page contract into a clean spreadsheet row, reliably, every time. Instead of asking the model to 'output JSON' and hoping (which fails about 15% of the time in production), you define exactly the shape you want as a tool schema, force the model to use that tool, and then validate every record before accepting it. When something is missing, you tell the model nullable fields are okay so it doesn't make data up. When the answer comes back wrong, the harness feeds the specific error back and asks again. The whole point is that extraction at scale needs deterministic shape AND honest gaps, not creative writing.
JSON Schema Definition
the contract, in input_schemaThe output shape lives inside a tool definition, not as freeform text instruction. Required vs nullable, enum vs string, integer vs number. Every property is explicit. The model can only emit a tool_use call that matches; the SDK rejects anything else.
Configurationtools = [{ name: "extract_record", input_schema: { type: "object", properties: { customer_id: { type: "string", pattern: "^cust_[0-9]+$" }, refund_amount: { type: ["number", "null"] }, refund_reason: { type: "string", enum: ["damage", "wrong_item", "late", "other", "unclear"] } }, required: ["customer_id", "refund_amount", "refund_reason"] } }]Concept: structured-outputs →Forced tool_choice
tool_choice: { type: 'tool', name: ... }Setting tool_choice to a specific tool name guarantees the model fires that tool. No prose wrapping, no 'I'd be happy to help' preamble, no probabilistic adherence. This is the single biggest reliability lever; it converts 85% prompt-only adherence into 100% structural adherence.
Configurationtool_choice: { type: 'tool', name: 'extract_record' }. Use 'auto' only for open-ended flows where the model decides whether to call any tool. Forced is for mandatory extraction.Concept: tool-choice →Validation-Retry Loop
parse → validate → feed-error-backSchema enforcement guarantees STRUCTURE. Semantic validation (date format, amount sign, ID pattern, business rules) runs in code after parse. On failure, the harness feeds a specific error message back to the model ('refund_amount is -50, must be ≥ 0') and retries. Typically converges in ≤ 2 retries. Generic 'try again' doesn't work; specific errors do.
Configurationloop: extract → parse → validate_semantically → if invalid, append { role: 'user', content: 'Validation failed: <specific error>. Re-extract.' } → retry. Max retries: 3. After 3, route to human review.Concept: evaluation →Nullable Fields + Enum Escape Hatches
the anti-fabrication architectureWhen a source genuinely doesn't contain a value, the model has two honest options: emit null (if the schema allows nullable) or emit a designated 'unclear' / 'not_provided' enum value. Without these escape hatches, required-string fields force the model to invent. Fabrication rate climbs above 5%. With them, fabrication drops below 1%.
ConfigurationField types: ["string", "null"] for optional values. Enums always include "unclear" or "other" as the last option. Few-shot examples explicitly show the model emitting unclear when source is silent. Anchors the behaviour.Concept: structured-outputs →
Schema Caching + Batch API
cost discipline at volumeThe schema is the largest stable token cost (~500-2000 tokens depending on complexity). Mark the tools array with cache_control: ephemeral; the 5-min TTL keeps it warm across sustained traffic, dropping schema-token cost ~90%. For overnight bulk runs, the Batch API gives a flat 50% discount. Combined with caching, bulk extraction cost drops 95%+ vs naive sync calls.
ConfigurationSync API: tools array with cache_control: { type: 'ephemeral' }. Cache hit rate ≥ 70% within 5-min windows. Batch API: submit 1000+ extractions overnight, results within 24h, no real-time retries (resubmit failures the next batch).Concept: prompt-caching →The problem
- Guaranteed structure on every output. A downstream pipeline must never see a record missing a field.
- Honest non-answers when source data is genuinely missing. Better an explicit 'unclear' than a fabricated value.
- Bulk extraction at acceptable cost. 1000 documents/night at <$5 total.
- Prompt 'output JSON' → 15% leak with prose wrapping ('Sure, here's the JSON:'); downstream parser breaks.
- Required fields with no nullable option → model fabricates values when source is silent (refund_reason becomes 'customer dissatisfied' even when the email said nothing).
- Single-pass extraction with no retry → semantic errors slip through (date as 'next Tuesday', amount as -50, customer_id with embedded whitespace).
- ✓ Schema conformance = 100% (forced tool_use guarantees shape)
- ✓ Fabrication rate < 1% (nullable + enum escapes give the model an honest opt-out)
- ✓ Validation-retry convergence ≥ 95% within 3 attempts; remainder routed to human review
- ✓ Bulk runs use Batch API (50% discount, 24h SLA) for non-blocking volume
- ✓ Schema cached with cache_control: ephemeral for ~90% savings on sustained traffic
One run, traced end to end
8 steps to production
Author the JSON schema as a tool definition
Define the output shape in tools[0].input_schema. A JSON Schema object. Every required field listed in required[]. Every optional field has ["<type>", "null"] so the model can emit null. Every constrained string is an enum with an explicit escape (unclear, not_provided, other). Add pattern regex on IDs that have a known format.
Concept: structured-outputs →from anthropic import Anthropic
client = Anthropic()
EXTRACT_TOOL = {
"name": "extract_record",
"description": "Extract a structured record from a customer email.",
"input_schema": {
"type": "object",
"properties": {
"customer_id": {"type": "string", "pattern": "^cust_[0-9]+$"},
"refund_amount": {"type": ["number", "null"]},
"refund_reason": {
"type": "string",
"enum": ["damage", "wrong_item", "late", "other", "unclear"],
},
"urgency": {
"type": "string",
"enum": ["low", "medium", "high", "unclear"],
},
},
"required": ["customer_id", "refund_amount", "refund_reason", "urgency"],
},
}Force tool_choice to the extraction tool
tool_choice: { type: 'tool', name: 'extract_record' } is the structural contract. The model has no choice but to fire the tool with arguments matching the schema. Any prose preamble or wrapping disappears. This single setting turns 85% prompt-only adherence into 100% structural adherence.
Concept: tool-choice →Add nullable types and enum escape hatches
Every field that might be genuinely missing in the source gets ["<type>", "null"]. Every constrained string includes an explicit unclear / not_provided / other option. This gives the model an honest exit when the source is silent. Without it, required-string fields force fabrication. Pair with a few-shot example showing the model correctly emitting unclear.
Concept: structured-outputs →Wrap extraction in a validation-retry loop
Schema guarantees structure; semantics need code. After parsing the tool_use input, validate semantically: refund_amount > 0, customer_id matches the canonical pattern beyond the regex, urgency-vs-amount sanity (a $50K refund tagged 'low' is suspicious). On failure, feed a specific error back to the model and retry. Specific errors converge; generic 'try again' loops forever.
Concept: evaluation →Cache the schema with cache_control: ephemeral
The schema is the largest stable token cost in steady-state extraction (~500-2000 tokens for non-trivial shapes). Mark the tools array with cache_control: { type: 'ephemeral' }. The 5-min TTL keeps it warm across sustained traffic; cached input tokens cost ~10% of fresh tokens. Hit rate stays ≥ 70% with continuous extraction; ~90% schema-token savings.
Concept: prompt-caching →Add a PreToolUse hook on policy bounds
When extraction touches policy-bearing values (refund cap, transaction limits), don't trust the model. Wrap it in a PreToolUse hook that exits 2 on violation. The hook reads tool_input.refund_amount and compares to the known cap; on breach, it returns an error to the model with the cap reference, and the model re-extracts with the constraint visible. Deterministic policy enforcement, not probabilistic.
Concept: hooks →Use Batch API for bulk overnight runs
Sync API is the right call when latency matters. For overnight backfills (1000+ documents), the Batch API gives a flat 50% discount with a 24h SLA. Combined with schema caching, bulk extraction cost drops 95%+ vs naive sync calls. Resubmit failures as a new batch the next morning. Batch API is async, no real-time retry inside the batch.
Concept: batch-api →Stratified accuracy reporting (not aggregate)
A 95% aggregate accuracy can hide a 60% accuracy on a critical document type. Track validation pass rate stratified by source. By document type (email vs PDF vs HTML), by sender domain, by extraction date. Bad strata surface fast. Aggregate metrics lie; stratified ones tell the truth.
Concept: evaluation →9 decisions the exam turns into distractors
Prompt instruction 'output JSON' or 'respond with valid JSON'
Forced tool_choice with input_schema as the contract
Required field with no nullable / no escape. Force the model to invent
["<type>", "null"] AND/OR enum with explicit 'unclear' option
Generic 'please try again' or single-pass with no retry
Validation-retry loop with the SPECIFIC error fed back
Sync API with caching, or sync API without caching
Batch API + schema caching (50% × ~90% = ~95% savings)
System prompt says 'respond with JSON only'. ~15% of responses include prose wrapping ('Sure, here's the JSON:'); downstream parser breaks on every fifth document.
Forced tool_choice + input_schema. The model has no choice but to fire the tool with arguments matching the schema. 100% structural adherence.
refund_reason is required string. Source email says nothing about a reason. Model fabricates 'customer dissatisfied' to satisfy the schema. Fabrication rate ~7%.
Make the field nullable AND/OR add an explicit 'unclear' enum option. Few-shot one example showing the model emit 'unclear' on a silent source. Fabrication drops below 1%.
Validation fails (refund_amount = -50). The pipeline drops the record entirely. Operator sees 5% silent loss; nobody notices for two weeks.
Validation-retry loop: feed the specific error back to the model, retry up to 3 times. 70-80% of validation failures converge within 2 retries.
Schema accepts refund_amount: -50 (number type passes). Schema accepts customer_id: 'cust_ 42' (string type passes). Bad data ships downstream.
Validate semantically AFTER schema parse: bounds checks (amount ≥ 0), regex on IDs (no embedded whitespace), business rules (urgency-vs-amount sanity). Schema enforces shape; code enforces meaning.
Refund cap is enforced in the system prompt ('never extract refund_amount > 500'). Production sees 3% violations leak through; auditor flags.
PreToolUse hook on extract_record: reads tool_input.refund_amount, compares to policy cap, exits 2 on breach with a specific error message. Deterministic, not probabilistic.
Cost & latency
Schema ~1500 tokens at cache-read price (~$0.0001) + email body ~500 input tokens + ~150 output tokens. Sustained traffic with ≥70% cache hit rate keeps per-record cost predictable.
5-10% of records retry once; 1-2% retry twice. Specific-error feedback converges quickly. Overall pipeline cost up ~5% to gain ~99% schema-conformance + ~99% semantic-conformance.
Batch API flat 50% discount × schema caching ~90% savings = ~95% total. 1000 documents @ ~$0.0008 each (sync cached) drops to ~$0.0004 each (batch + cache). $0.40 vs $0.80 per 1000.
PreToolUse hook is a Python/TS subprocess reading stdin and exiting 0/2. No LLM call. Cost is purely syscall-level latency.
Real production extraction at typical complexity. Batch + cache halves it. Adding human review of unconverged records adds operator-time cost but recovers the long tail.
Check every gate before release
5 exam-pattern questions
Work through one question at a time, check the architecture, then move through the set.
A support conversation hits turn 30 with progressive summarization enabled, and the agent suddenly cannot recall the customer ID. What is missing from the architecture?
Frequently asked
Why not just prompt 'output JSON'. Claude is good at it now?
Probabilistic ≠ guaranteed. Even at 95% adherence, a 5% prose-wrapped output rate is one broken record per 20 documents. Forced tool_choice is structural. The SDK rejects anything that doesn't match the schema. The cost is identical; the reliability difference is decisive. Use prompts for tone; use forced tools for shape.
Can I cache the schema if it changes between calls?
Cache the stable parts. If the schema body is fixed but a few enum values vary by tenant, split the tools array: stable common schema (cached) + small per-tenant additions (fresh). Cache only what's stable across calls; the cache key is sensitive to byte-level changes.
How does Batch API interact with the validation-retry loop?
Batch is async. No inside-the-batch retry. Submit, wait 24h, harvest results. Validate each; if some fail, submit those failures (with the specific error in the next message) as a NEW batch. Most converge in batch-2. For records that need real-time retry, route them to the sync pipeline.
What's the difference between the schema's pattern regex and code-level validation?
Schema validates shape; code validates meaning. pattern: '^cust_[0-9]+$' rejects malformed IDs at parse time. Faster, structural. Semantic checks (the customer_id must exist in our DB, the refund_amount must be ≤ the original purchase total) need code; they're business rules, not syntax. Use pattern for cheap structural rejection; use code for everything that requires lookups or business logic.
Should I run extended_thinking with structured extraction?
Generally no. They're incompatible with forced tool_choice. When the model needs to reason about ambiguous source text, set tool_choice: 'auto' and accept ~95% reliability, OR run a sync pre-pass to disambiguate, then a forced extraction on the cleaned input. Don't try to combine extended_thinking with forced tool_choice in one call.
How do I test that nullable + enum escapes are working?
Adversarial test set. Compose 50 short inputs where the value is GENUINELY missing or ambiguous. Run extraction. The right behaviour is a mix of null and 'unclear'. Never invented values. If you see invented values (e.g. a refund_reason that's not in the source), the few-shot or the schema doesn't yet give the model an honest exit. Iterate.
Does forced tool_choice work with multi-tool registries?
Yes. Tool_choice picks one specific tool by name. Even in a 5-tool registry, tool_choice: { type: 'tool', name: 'extract_record' } forces THAT tool. The other 4 are inert for this call. Use this when extraction is mandatory but the broader agent has other tools available; for extraction-only, the registry can have just one tool.
When do I use Batch API vs sync vs caching for cost optimization?
Three-axis decision. (1) Latency-sensitive (interactive review, real-time agent loops): sync API + prompt-cache on the schema and stable system prompt. ~80% off the cached portion at >=70% hit rate. (2) Latency-tolerant bulk (>=100 docs overnight): Batch API for the flat 50% discount, 24h SLA. Combine with caching only at the per-batch sub-batch level (ephemeral cache is per-request in Batch). (3) Mixed traffic: sync the long tail, batch the predictable backlog. Cross-references: P3.5 uses Batch API for nightly audits; P3.8 uses Batch API for bulk long-doc extraction; P3.7 covers the 4-line tool-description pattern that keeps cached tools array stable. Tagged related: cost-optimization cluster.
Why does the model's self-reported confidence not gate routing?
Calibration anti-pattern. Models emit confidence scores that correlate weakly with actual correctness, especially on out-of-distribution inputs. A 0.95 confidence on a fabricated value still ships fabricated data. The right architecture is structural: validation-retry on schema + semantic checks (sum equals total, currency in enum, date sanity), and stratified accuracy reporting by document type. Use confidence as a soft signal in routing decisions (escalate to human if confidence < 0.5) but never as the primary gate. Tagged related: evaluation-and-evals cluster.