# Production Claude Chat Application

> A streaming chat backend on the Messages API. The server owns conversation history (the API is stateless), renders content_block_delta events to the client as they arrive, appends the full response.content (not just the text) back onto history so tool/thinking blocks survive, and manages the context window with token counting plus compaction rather than an unbounded array. Prompt caching on the system prompt cuts the repeated-context cost that dominates long conversations.

**Sub-marker:** CCD-F.1
**Domains:** CCDVF-D2 · Applications & Integration, CCDVF-D6 · Prompt & Context Engineering, CCDVF-D5 · Model Selection & Optimization
**Exam weight:** CCD-F D2 (33% - the heaviest domain)
**Build time:** 20 minutes
**Source:** 🟡 Developer-cert production pattern
**Canonical:** https://claudearchitectcertification.com/scenarios/production-claude-chat-application
**Last reviewed:** 2026-07-12

## In plain English

This is the app most developers build first: a chat UI backed by Claude, with responses that stream in word by word instead of appearing all at once after a long wait. It exists because a 6-second pause before any text appears feels broken to a user, while the same 6 seconds spent watching words stream in feels fast. The hard part isn't calling the API, it's everything around the call: keeping the conversation history somewhere durable, knowing when the context window is about to overflow, and making sure a client disconnect mid-stream doesn't silently lose the assistant's answer.

## Exam impact

Applications & Integration (D2) is 33.1% of CCD-F, the single heaviest domain on the whole exam. This scenario is the canonical D2 build: a real streaming integration, not a toy single-shot call. Model Selection (D5, 16.8%) and Prompt & Context Engineering (D6, 11%) both ride through the same code path.

## The problem

### What the customer needs
- See the response start appearing within ~200ms, not after the full generation finishes.
- Never lose an in-progress answer when the browser tab closes or the network blips.
- Keep a 40-turn conversation coherent without the context window silently overflowing.

### Why naive approaches fail
- Client-side-only history means refreshing the page loses the whole conversation (the Messages API is stateless - nothing is stored server-side unless you store it).
- Appending only response.text back onto history drops tool_use and thinking blocks, corrupting the next turn's context.
- An unbounded messages array eventually exceeds the context window and every request starts failing at once instead of degrading gracefully.

### Definition of done
- Time-to-first-token < 300ms on a warm cache
- Zero conversations lost on client disconnect (server-side persistence, not client memory)
- Context budget enforced before the 1M-token ceiling, with graceful compaction
- Full response.content persisted per turn, including non-text blocks

## Concepts in play

- 🟢 **Streaming** (`streaming`), SSE event handling + UI rendering
- 🟢 **Session state** (`session-state`), Server-owned conversation history
- 🟡 **Context window management** (`context-window`), Token budget + compaction
- 🟢 **Prompt caching** (`prompt-caching`), System prompt cache breakpoint
- 🟡 **Model selection** (`model-selection-tradeoffs`), Default tier for chat traffic

## Components

### Stream Handler, SSE consumer - UI renderer

Opens client.messages.stream(), listens for text deltas, and writes them to the client connection as they arrive instead of buffering the whole response.

**Configuration:** client.messages.stream({model, max_tokens, system, messages}) with a text-delta listener; call .finalMessage() once to get the complete Message for persistence.
**Concept:** `streaming`

### Session Store, server-owned history

Persists the full message array per session_id in a database. The API is stateless - every request resends the whole history, so this store is the only source of truth.

**Configuration:** sessions[session_id].messages.push({role: 'assistant', content: response.content}) - the full content array, never just the extracted text.
**Concept:** `session-state`

### Context Budget Manager, count_tokens + compaction

Calls messages.count_tokens() before each turn against the live model. When the count nears the working budget, trims oldest turns or engages compaction rather than sending an oversized request.

**Configuration:** if count_tokens(messages).input_tokens > BUDGET: trim oldest non-system turns, or (beta) enable context_management compaction with beta compact-2026-01-12.
**Concept:** `context-window`

### Cache Breakpoint, system prompt - ephemeral

Marks the last block of the (stable, non-interpolated) system prompt with cache_control so repeated turns in the same conversation read from cache instead of reprocessing the full prefix.

**Configuration:** system: [{type: "text", text: SYSTEM_PROMPT, cache_control: {type: "ephemeral"}}] - SYSTEM_PROMPT never has a timestamp or session id interpolated into it.
**Concept:** `prompt-caching`

## Build steps

### 1. Open a stream and render deltas

Use the SDK's stream() helper rather than a manual SSE parser. Only text_delta events touch the UI; other event types drive server-side bookkeeping.

**Python:**

```python
client = Anthropic()

def stream_turn(messages, system_prompt):
    with client.messages.stream(
        model="claude-sonnet-5",
        max_tokens=4096,
        system=[{"type": "text", "text": system_prompt,
                 "cache_control": {"type": "ephemeral"}}],
        messages=messages,
    ) as stream:
        for text in stream.text_stream:
            yield text  # forward to the client connection
        return stream.get_final_message()
```

**TypeScript:**

```typescript
const client = new Anthropic();

async function streamTurn(messages: Anthropic.MessageParam[], systemPrompt: string) {
  const stream = client.messages.stream({
    model: "claude-sonnet-5",
    max_tokens: 4096,
    system: [{ type: "text", text: systemPrompt, cache_control: { type: "ephemeral" } }],
    messages,
  });
  stream.on("text", (delta) => sendToClient(delta));
  return await stream.finalMessage();
}
```

Concept: `streaming`

### 2. Persist the full content array, not the text

The final assistant turn must carry every content block, including thinking or tool_use blocks, or the next request loses structure the API needs to stay coherent.

**Python:**

```python
def persist_turn(session_id: str, user_text: str, final_message):
    store.append(session_id, {"role": "user", "content": user_text})
    store.append(session_id, {"role": "assistant", "content": final_message.content})
    # NOT final_message.content[0].text - that drops non-text blocks
```

**TypeScript:**

```typescript
async function persistTurn(sessionId: string, userText: string, finalMessage: Anthropic.Message) {
  await store.append(sessionId, { role: "user", content: userText });
  await store.append(sessionId, { role: "assistant", content: finalMessage.content });
  // NOT finalMessage.content[0].text - that drops non-text blocks
}
```

Concept: `session-state`

### 3. Budget the context window before every turn

Call count_tokens against the model you're about to use (token counts are model-specific) and trim or compact before the request, not after it fails.

**Python:**

```python
def enforce_budget(messages, system_prompt, model="claude-sonnet-5", budget=180_000):
    counted = client.messages.count_tokens(
        model=model, system=system_prompt, messages=messages,
    )
    while counted.input_tokens > budget and len(messages) > 2:
        messages = messages[2:]  # drop oldest user/assistant pair
        counted = client.messages.count_tokens(
            model=model, system=system_prompt, messages=messages,
        )
    return messages
```

**TypeScript:**

```typescript
async function enforceBudget(
  messages: Anthropic.MessageParam[],
  systemPrompt: string,
  model = "claude-sonnet-5",
  budget = 180_000,
) {
  let counted = await client.messages.countTokens({ model, system: systemPrompt, messages });
  while (counted.input_tokens > budget && messages.length > 2) {
    messages = messages.slice(2); // drop oldest user/assistant pair
    counted = await client.messages.countTokens({ model, system: systemPrompt, messages });
  }
  return messages;
}
```

Concept: `context-window`

### 4. Branch on stop_reason after the stream ends

The stream's final message carries the same stop_reason contract as a non-streaming call. Handle max_tokens and refusal explicitly instead of assuming every stream ends cleanly.

**Python:**

```python
final = stream.get_final_message()
if final.stop_reason == "max_tokens":
    mark_truncated(session_id)
elif final.stop_reason == "refusal":
    log_refusal(session_id, final.stop_details)
elif final.stop_reason == "end_turn":
    persist_turn(session_id, user_text, final)
```

**TypeScript:**

```typescript
const final = await stream.finalMessage();
if (final.stop_reason === "max_tokens") {
  markTruncated(sessionId);
} else if (final.stop_reason === "refusal") {
  logRefusal(sessionId, final.stop_details);
} else if (final.stop_reason === "end_turn") {
  await persistTurn(sessionId, userText, final);
}
```

Concept: `streaming`

### 5. Inject runtime context without breaking the cache

On models that support it, append a role: 'system' message to the array for mid-conversation operator context instead of rewriting the top-level system prompt, which would invalidate the cached prefix on every turn.

**Python:**

```python
def inject_runtime_context(messages, note: str):
    # Claude Opus 4.8 only, no beta header. Must follow a user turn.
    messages.append({"role": "system", "content": note})
    return messages
```

**TypeScript:**

```typescript
function injectRuntimeContext(messages: Anthropic.MessageParam[], note: string) {
  // Claude Opus 4.8 only, no beta header. Must follow a user turn.
  messages.push({ role: "system", content: note });
  return messages;
}
```

Concept: `prompt-caching`

### 6. Handle client disconnect without losing the turn

The Messages API stream itself has no server-side replay. Keep the request running server-side even if the client connection drops, and persist on completion regardless of whether anyone was still listening.

**Python:**

```python
async def run_turn_resilient(session_id, user_text):
    messages = load_history(session_id) + [{"role": "user", "content": user_text}]
    try:
        final = await run_stream(messages)  # keeps running server-side
    finally:
        if final:
            persist_turn(session_id, user_text, final)  # persists even if client left
```

**TypeScript:**

```typescript
async function runTurnResilient(sessionId: string, userText: string) {
  const messages = [...(await loadHistory(sessionId)), { role: "user" as const, content: userText }];
  let final: Anthropic.Message | undefined;
  try {
    final = await runStream(messages); // keeps running server-side
  } finally {
    if (final) await persistTurn(sessionId, userText, final); // persists even if client left
  }
}
```

Concept: `session-state`

## Decision matrix

| Decision | Right answer | Wrong answer | Why |
|---|---|---|---|
| Where conversation history lives | server-side session store, keyed by session_id | client-side only (localStorage/component state) | The Messages API is stateless. Client-only history loses everything on refresh and can't be trusted for gating or audit. |
| What gets persisted per turn | the full response.content array | just the extracted text string | Dropping tool_use/thinking blocks corrupts the structure the next request needs, even in a text-only chat app that later adds tools. |
| Context overflow handling | count_tokens before the request, trim/compact proactively | let the request fail, then handle the error | A failed request after the user has waited is a worse experience than a quiet, proactive trim. |
| Default model tier for chat traffic | claude-sonnet-5 | claude-opus-4-8 for every message | Sonnet 5 is the cost/quality default for conversational traffic; reserve Opus for a router to escalate into (Scenario CCD-F.4), not the default path. |

## Failure modes

| Anti-pattern | Failure | Fix |
|---|---|---|
| CCDF-01 · Content truncation on persist | Server stores finalMessage.content[0].text as the assistant turn. | Store the full content array. Non-text blocks are silently dropped otherwise, breaking later turns. |
| CCDF-02 · Cache-breaking interpolation | System prompt string includes f"Current time: {now()}" at the top. | Keep the system prompt byte-identical across requests. Inject dynamic context via a role: 'system' message instead. |
| CCDF-03 · No proactive context budget | Messages array grows unbounded until a request 400s on context length. | Call count_tokens before every request and trim or compact before it fails, not after. |
| CCDF-04 · Streaming with no persistence path | Assistant text is only ever rendered to the open connection, never saved. | Always call .finalMessage() / get_final_message() and persist it, independent of whether the client is still connected. |

## Implementation checklist

- [ ] Server-side session store keyed by session_id (`session-state`)
- [ ] Stream text deltas to the client; persist full content array on completion (`streaming`)
- [ ] count_tokens budget check before every request (`context-window`)
- [ ] System prompt is byte-stable; cache_control on its last block (`prompt-caching`)
- [ ] stop_reason branch covers end_turn, max_tokens, and refusal
- [ ] Default model is Sonnet 5, not Opus, for standard chat traffic
- [ ] Client disconnect does not abort server-side persistence

## Cost &amp; latency

- **Per-turn tokens (typical):** ~1,800 input (cached) - 600 output, System prompt + tool defs read from cache after turn 1; only new user text + growing tail cost full price.
- **Per-turn cost (Sonnet 5):** ~$0.006, At $2/$10 intro per MTok with ~70% cache-read rate on the stable prefix.
- **Time-to-first-token:** ~200-300ms, Streaming avoids waiting for the full generation; first token arrives as soon as the model emits it.

## Domain weights

- **CCDVF-D2 · Applications & Integration (33.1%):** Stream Handler + Session Store
- **CCDVF-D6 · Prompt & Context Engineering (11.0%):** Cache Breakpoint + system prompt design
- **CCDVF-D5 · Model Selection & Optimization (16.8%):** Default model tier selection

## Practice questions

### Q1. Your chat app persists response.content[0].text as the assistant turn. After adding a tool, multi-turn conversations start behaving inconsistently. Why?

Persisting only the text block drops any tool_use (or thinking) blocks that were also in the response. Persist the full content array every time, regardless of whether the current turn used tools.

### Q2. Users report the assistant sometimes 'restarts' mid-explanation on long conversations. What's the likely cause and fix?

The messages array is growing unbounded and hitting the context window, silently changing what the model can see. Add a proactive count_tokens budget check and trim or compact before the request, not after a failure.

### Q3. You interpolate the current session ID into the system prompt for logging. Cache hit rate is near zero. What's wrong?

Prompt caching is a byte-exact prefix match. A per-session value in the system prompt makes every request's prefix unique. Move that value out of the cached system prompt and into a later message, or use a role: 'system' mid-conversation message.

### Q4. A user's browser tab closes mid-response. Should the server abort the in-flight request?

No. Keep the request running server-side and persist the final message on completion regardless of client connection state. The Messages API has no server-side replay of a dropped stream, so losing the in-flight request also loses the turn.

## FAQ

### Q1. Do I need a database, or can I keep history in memory?

Use a database (or Redis with persistence) for anything beyond a local demo. In-memory history is lost on process restart and doesn't scale past one server instance.

### Q2. Should I stream on every request?

Stream whenever max_tokens is large or the user is waiting live for the response - which is essentially always true for a chat UI. Non-streaming is fine for short, backend-only calls with small max_tokens.

### Q3. How do I know when to trim history versus use compaction?

Trimming (dropping oldest turns) is simple and works for most chat apps. The beta compaction feature (compact-2026-01-12) is worth adopting when you need the model to retain awareness of dropped history rather than losing it outright - useful for long support or coaching sessions.

### Q4. What model should a general-purpose chat app default to?

Claude Sonnet 5 for the vast majority of conversational traffic. It is the best cost/quality balance in the current lineup; reserve Opus 4.8 for a router-driven escalation path (see the model-selection scenario), not as the default.

### Q5. Is it safe to cache the entire conversation, not just the system prompt?

The message array is not typically cached because it grows every turn and each growth invalidates a naive full-array cache marker. Cache the stable prefix (system + tool defs); let the growing message tail be uncached, or use auto top-level caching which places the marker on the last cacheable block.

## Production readiness

- [ ] Load test streaming under concurrent connections
- [ ] Verify persistence completes even when the client disconnects mid-stream
- [ ] Alert on time-to-first-token p95 exceeding target
- [ ] Alert on cache_read_input_tokens dropping to zero (silent cache invalidation)
- [ ] Context-budget trim logic covered by a test with an oversized synthetic history
- [ ] stop_reason handling covers refusal, not just end_turn/max_tokens

---

**Source:** https://claudearchitectcertification.com/scenarios/production-claude-chat-application
**Last reviewed:** 2026-07-12

**Evidence tiers**, 🟢 official Anthropic doc · 🟡 partial doc / inferred · 🟠 community-derived · 🔴 disputed.