AI Application Security: Prompt Injection & Untrusted Input.

Anthropic's guardrails documentation splits AI application security into two threat models with genuinely different mitigations. Jailbreaks / direct prompt injection: the app's own user is the adversary, crafting inputs designed to bypass Claude's instructions or safety behavior. Indirect prompt injection: the user is trusted, but Claude processes third-party content, a web page, an email, a document, a tool result, that itself carries adversarial instructions the user never wrote and may not even see. Treating these as one problem is a mistake this domain tests directly: direct injection needs input screening on what the user sends; indirect injection needs structuring around what Claude reads back from tools.

Agentic Claude applications need API-level controls for both threat models, plus separate data-retention and PII controls at the platform layer. Authentication, authorization, confidentiality, and integrity are related but distinct concerns from injection defense itself, and they matter precisely because prompt-level defenses are probabilistic, not a hard boundary: authorization here means least-privilege tool scoping (an injected instruction can only reach whatever the tool's own credentials and permissions allow) plus action authorization - a human-approval or policy gate in front of any high-risk tool call (send email, transfer funds, delete data), enforced regardless of whether the content was flagged as suspicious. Confidentiality and integrity mean minimal secret exposure to tools that read untrusted content, and validating tool output before it is trusted or acted on. An app can screen every user message perfectly and still leak data through an unstructured tool result, or the reverse - which is why the real security boundary sits at the tool-execution layer (what a call is allowed to do and who approves it), not in how the untrusted text is formatted.

Direct-injection / jailbreak defenses operate on the user's own input. A harmlessness screen, typically a lightweight model like Claude Haiku 4.5, pre-classifies user input via output_config with a json_schema format forcing a boolean like is_harmful, before the main call runs. Input-validation filters catch known injection patterns, hardened system prompts state explicit refusal instructions, and repeat offenders get throttled or banned.

Indirect-injection structuring calibrates the model's trust; it is not itself a security boundary. Untrusted content belongs only inside tool_result blocks, never in system or plain user text, alongside an explicit <untrusted_content_policy> stated in the system prompt - Claude is trained to treat instructions appearing inside tool results with more skepticism, which lowers the probability it acts on them. Third-party strings get JSON-encoded, which prevents quote or tag characters in the retrieved content from breaking out of the data structure into instruction syntax - a narrow, mechanical fix for one escaping vector, not a guarantee the model ignores the content's meaning. Critically, your own instructions never go inside a `tool_result`, send them in the following user turn, or via a mid-conversation system message on Opus 4.8+, since a tool result is data Claude reads, not a channel Claude should take orders from. None of this placement or encoding work replaces enforcement - it reduces how often an injection succeeds, it does not cap what one can do if it does.

The actual enforcement boundary is least privilege plus action authorization, backed by output screening. Least privilege scopes what each tool's own credentials and permissions allow, so a successful injection can only reach what that specific tool was already permitted to touch - sandboxed execution and minimal secret access limit blast radius even when the model is fooled. Action authorization gates the tool call itself: high-risk actions (sending data externally, financial transfers, deletions) sit behind a human-approval or policy check (an always_ask permission policy or a PreToolUse hook) that fires regardless of whether any classifier flagged the content as suspicious - this is the hard boundary content-formatting can't provide. A Haiku classifier screening raw tool output for injection_suspected: boolean before it is ever returned as a tool_result is a further, independent check, not a substitute for the permission gate. The handle-tool-calls doc reinforces the same principle from the tool-design side: tool results carry content from sources outside your control and must be treated as untrusted, with is_error: true reserved for genuine execution failures.

Anthropic also hardens the model itself, but frames this as a floor, not a ceiling. Claude is trained via RL against injections embedded in simulated web content, rewarding refusal, and runs classifiers scanning all untrusted content entering the context window, hidden text, manipulated images, deceptive UI. Computer-use adds screenshot classifiers that trigger user-confirmation prompts. Anthropic reported roughly a 1% attack success rate for Opus 4.5 in browser use (100 attempts per environment, adaptive attackers), low enough to inform moving Claude for Chrome from research preview to beta, but not zero, which is exactly why app-level defenses stay mandatory rather than optional.

Data retention and PII handling sit at the platform layer, separate from injection defense. Conversation content is not retained by default; Zero Data Retention (ZDR) is opt-in per-organization via sales, never an automatic state. Covered Models require 30-day retention, and ZDR is unavailable for them. HIPAA readiness is a separate, broader control set (encryption, access controls, audit logging) for PHI, not a fast-deletion guarantee. Structured-output schemas with strict: true are cached separately from message content and must never contain PHI, and flagged content may be retained up to 2 years regardless of an org's retention arrangement.

Authentication and authorization are load-bearing here, not a side concern. For this objective, authorization is what stops an indirect injection from mattering even when the model is fooled: scope each tool's credentials to the minimum it needs (a document-fetch tool gets no write access; an email-summarizer never holds a send-mail credential), and require action authorization - human or policy approval - in front of any tool call with real-world consequence, independent of any injection screen. Confidentiality means an untrusted-content-reading tool never carries a secret it doesn't need; integrity means tool output is screened before the agent (or a downstream system) trusts or acts on it. The transport- and identity-level mechanics behind this - MCP OAuth requirements, Agent SDK canUseTool/PreToolUse hooks, per-tool permission policies - are covered in depth on the auth and access control concept page, see Study Next.