The problem
What the customer needs
- See the response start appearing within ~200ms, not after the full generation finishes.
- Never lose an in-progress answer when the browser tab closes or the network blips.
- Keep a 40-turn conversation coherent without the context window silently overflowing.
Why naive approaches fail
- Client-side-only history means refreshing the page loses the whole conversation (the Messages API is stateless - nothing is stored server-side unless you store it).
- Appending only
response.textback onto history drops tool_use and thinking blocks, corrupting the next turn's context. - An unbounded messages array eventually exceeds the context window and every request starts failing at once instead of degrading gracefully.
- Time-to-first-token < 300ms on a warm cache
- Zero conversations lost on client disconnect (server-side persistence, not client memory)
- Context budget enforced before the 1M-token ceiling, with graceful compaction
- Full
response.contentpersisted per turn, including non-text blocks
The system
What each part does
4 components, each owns a concept. Click any card to drill into the underlying primitive.
Stream Handler
SSE consumer - UI renderer
Opens client.messages.stream(), listens for text deltas, and writes them to the client connection as they arrive instead of buffering the whole response.
Configuration
client.messages.stream({model, max_tokens, system, messages}) with a text-delta listener; call .finalMessage() once to get the complete Message for persistence.
Session Store
server-owned history
Persists the full message array per session_id in a database. The API is stateless - every request resends the whole history, so this store is the only source of truth.
Configuration
sessions[session_id].messages.push({role: 'assistant', content: response.content}) - the full content array, never just the extracted text.
Context Budget Manager
count_tokens + compaction
Calls messages.count_tokens() before each turn against the live model. When the count nears the working budget, trims oldest turns or engages compaction rather than sending an oversized request.
Configuration
if count_tokens(messages).input_tokens > BUDGET: trim oldest non-system turns, or (beta) enable context_management compaction with beta compact-2026-01-12.
Cache Breakpoint
system prompt - ephemeral
Marks the last block of the (stable, non-interpolated) system prompt with cache_control so repeated turns in the same conversation read from cache instead of reprocessing the full prefix.
Configuration
system: [{type: "text", text: SYSTEM_PROMPT, cache_control: {type: "ephemeral"}}] - SYSTEM_PROMPT never has a timestamp or session id interpolated into it.
Data flow
6 steps to production
Open a stream and render deltas
Use the SDK's stream() helper rather than a manual SSE parser. Only text_delta events touch the UI; other event types drive server-side bookkeeping.
client = Anthropic()
def stream_turn(messages, system_prompt):
with client.messages.stream(
model="claude-sonnet-5",
max_tokens=4096,
system=[{"type": "text", "text": system_prompt,
"cache_control": {"type": "ephemeral"}}],
messages=messages,
) as stream:
for text in stream.text_stream:
yield text # forward to the client connection
return stream.get_final_message()Persist the full content array, not the text
The final assistant turn must carry every content block, including thinking or tool_use blocks, or the next request loses structure the API needs to stay coherent.
def persist_turn(session_id: str, user_text: str, final_message):
store.append(session_id, {"role": "user", "content": user_text})
store.append(session_id, {"role": "assistant", "content": final_message.content})
# NOT final_message.content[0].text - that drops non-text blocksBudget the context window before every turn
Call count_tokens against the model you're about to use (token counts are model-specific) and trim or compact before the request, not after it fails.
def enforce_budget(messages, system_prompt, model="claude-sonnet-5", budget=180_000):
counted = client.messages.count_tokens(
model=model, system=system_prompt, messages=messages,
)
while counted.input_tokens > budget and len(messages) > 2:
messages = messages[2:] # drop oldest user/assistant pair
counted = client.messages.count_tokens(
model=model, system=system_prompt, messages=messages,
)
return messagesBranch on stop_reason after the stream ends
The stream's final message carries the same stop_reason contract as a non-streaming call. Handle max_tokens and refusal explicitly instead of assuming every stream ends cleanly.
final = stream.get_final_message()
if final.stop_reason == "max_tokens":
mark_truncated(session_id)
elif final.stop_reason == "refusal":
log_refusal(session_id, final.stop_details)
elif final.stop_reason == "end_turn":
persist_turn(session_id, user_text, final)Inject runtime context without breaking the cache
On models that support it, append a role: 'system' message to the array for mid-conversation operator context instead of rewriting the top-level system prompt, which would invalidate the cached prefix on every turn.
def inject_runtime_context(messages, note: str):
# Claude Opus 4.8 only, no beta header. Must follow a user turn.
messages.append({"role": "system", "content": note})
return messagesHandle client disconnect without losing the turn
The Messages API stream itself has no server-side replay. Keep the request running server-side even if the client connection drops, and persist on completion regardless of whether anyone was still listening.
async def run_turn_resilient(session_id, user_text):
messages = load_history(session_id) + [{"role": "user", "content": user_text}]
try:
final = await run_stream(messages) # keeps running server-side
finally:
if final:
persist_turn(session_id, user_text, final) # persists even if client leftThe four decisions
| Decision | Right answer | Wrong answer | Why |
|---|---|---|---|
| Where conversation history lives | server-side session store, keyed by session_id | client-side only (localStorage/component state) | The Messages API is stateless. Client-only history loses everything on refresh and can't be trusted for gating or audit. |
| What gets persisted per turn | the full response.content array | just the extracted text string | Dropping tool_use/thinking blocks corrupts the structure the next request needs, even in a text-only chat app that later adds tools. |
| Context overflow handling | count_tokens before the request, trim/compact proactively | let the request fail, then handle the error | A failed request after the user has waited is a worse experience than a quiet, proactive trim. |
| Default model tier for chat traffic | claude-sonnet-5 | claude-opus-4-8 for every message | Sonnet 5 is the cost/quality default for conversational traffic; reserve Opus for a router to escalate into (Scenario CCD-F.4), not the default path. |
Where it breaks
4 failure pairs. Each maps to an exam-style question - the naive move on the left, the disciplined fix on the right.
Server stores finalMessage.content[0].text as the assistant turn.
Store the full content array. Non-text blocks are silently dropped otherwise, breaking later turns.
System prompt string includes f"Current time: {now()}" at the top.
Keep the system prompt byte-identical across requests. Inject dynamic context via a role: 'system' message instead.
Messages array grows unbounded until a request 400s on context length.
CCDF-03Call count_tokens before every request and trim or compact before it fails, not after.
Assistant text is only ever rendered to the open connection, never saved.
CCDF-04Always call .finalMessage() / get_final_message() and persist it, independent of whether the client is still connected.
Cost & latency
System prompt + tool defs read from cache after turn 1; only new user text + growing tail cost full price.
At $2/$10 intro per MTok with ~70% cache-read rate on the stable prefix.
Streaming avoids waiting for the full generation; first token arrives as soon as the model emits it.
Ship checklist
Two passes. Build-time gates verify the code; run-time gates verify the system in production.
Build-time
- Server-side session store keyed by session_id↗ session-state
- Stream text deltas to the client; persist full content array on completion↗ streaming
- count_tokens budget check before every request↗ context-window
- System prompt is byte-stable; cache_control on its last block↗ prompt-caching
- stop_reason branch covers end_turn, max_tokens, and refusal
- Default model is Sonnet 5, not Opus, for standard chat traffic
- Client disconnect does not abort server-side persistence
Run-time
- Load test streaming under concurrent connections
- Verify persistence completes even when the client disconnects mid-stream
- Alert on time-to-first-token p95 exceeding target
- Alert on cache_read_input_tokens dropping to zero (silent cache invalidation)
- Context-budget trim logic covered by a test with an oversized synthetic history
- stop_reason handling covers refusal, not just end_turn/max_tokens
Five exam-pattern questions
Your chat app persists `response.content[0].text` as the assistant turn. After adding a tool, multi-turn conversations start behaving inconsistently. Why?
tool_use (or thinking) blocks that were also in the response. Persist the full `content` array every time, regardless of whether the current turn used tools.Users report the assistant sometimes 'restarts' mid-explanation on long conversations. What's the likely cause and fix?
count_tokens budget check and trim or compact before the request, not after a failure.You interpolate the current session ID into the system prompt for logging. Cache hit rate is near zero. What's wrong?
role: 'system' mid-conversation message.A user's browser tab closes mid-response. Should the server abort the in-flight request?
Frequently asked
Do I need a database, or can I keep history in memory?
Should I stream on every request?
max_tokens is large or the user is waiting live for the response - which is essentially always true for a chat UI. Non-streaming is fine for short, backend-only calls with small max_tokens.How do I know when to trim history versus use compaction?
compact-2026-01-12) is worth adopting when you need the model to retain awareness of dropped history rather than losing it outright - useful for long support or coaching sessions.