Claude for Operations.

The webhook receiver projects raw monitoring payloads (Datadog/Prometheus/Sentry/custom) into a stable shape: {bucket, service, severity, root_cause_signal, retryable, raw}. The bucket is the single most important field. It's what the agent's routing logic branches on. Bucket assignment is rule-based for known patterns; ambiguous alerts get classified by Haiku (cheap, fast) and emit the bucket + a confidence score.

from enum import Enum
from typing import TypedDict, Literal

class Bucket(str, Enum):
    TRANSIENT  = "Transient"   # network blip, rate limit. Retry
    PERMISSION = "Permission"  # 403/401. Escalate, won't fix itself
    DATA       = "Data"        # malformed input. Surface to user
    BUSINESS   = "Business"    # policy violation. Block + log + escalate

class Alert(TypedDict):
    bucket: Bucket
    service: str
    severity: Literal["P1", "P2", "P3"]
    root_cause_signal: str
    retryable: bool
    raw: dict

def classify_alert(raw: dict) -> Alert:
    """Project arbitrary monitoring payload into the 4-bucket contract."""
    msg = (raw.get("message") or "").lower()
    if any(s in msg for s in ["503", "timeout", "rate limit", "circuit breaker"]):
        bucket, retryable = Bucket.TRANSIENT, True
    elif any(s in msg for s in ["403", "401", "forbidden", "unauthorized"]):
        bucket, retryable = Bucket.PERMISSION, False
    elif any(s in msg for s in ["malformed", "schema", "validation", "400"]):
        bucket, retryable = Bucket.DATA, False
    elif any(s in msg for s in ["policy", "compliance", "limit exceeded"]):
        bucket, retryable = Bucket.BUSINESS, False
    else:
        # Ambiguous. Classify with Haiku (cheap) and trust the bucket it returns
        bucket, retryable = haiku_classify_bucket(raw)
    return {
        "bucket": bucket,
        "service": raw.get("service", "unknown"),
        "severity": raw.get("severity", "P2"),
        "root_cause_signal": raw.get("root_cause") or msg[:100],
        "retryable": retryable,
        "raw": raw,
    }

def haiku_classify_bucket(raw: dict) -> tuple[Bucket, bool]:
    """Cheap fallback classifier for ambiguous alerts."""
    resp = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=64,
        messages=[{"role": "user", "content":
            f"Classify into one bucket (Transient|Permission|Data|Business) "
            f"+ retryable (true|false). Alert: {raw}\n"
            f"Output JSON only: {{\"bucket\": ..., \"retryable\": ...}}"}],
    )
    parsed = json.loads(resp.content[0].text)
    return Bucket(parsed["bucket"]), parsed["retryable"]

Every runbook is a named sequence of safe, reversible commands with explicit preconditions, timeouts, and rollback steps. The registry stays at 3-5 entries. The agent's routing accuracy degrades past 5, and a sixth runbook for an edge case is exactly the wrong move (escalate the edge case instead). Each runbook is reviewed in PRs and deployed through CI like application code.

The hook is the deterministic safety gate. It reads tool_name + tool_input.command from stdin JSON, applies a regex blocklist (compiled once), exits 2 with a stderr message on match. The model sees the deny as a tool_result with is_error: true and re-plans. No prompt-injection bypass; the blocklist is in code, not in the prompt.

When the agent can't act safely. Unknown runbook, hook denied, ambiguous alert, explicit escalation request. The harness emits a structured escalation block to PagerDuty / Slack. Six required fields. Humans triage in ~30 seconds vs ~5 minutes reading a raw transcript. The block is the single most important UX artifact in the whole system; tune the wording to be a one-screen read.

PostToolUse fires AFTER every tool call (Bash, runbook, escalate, log query, hook decisions). Writes a canonical row: ts, tool_name, tool_input, tool_result, latency_ms, hook_decisions, incident_id. Append-only JSONL on durable storage; retain ≥ 90 days. Indexed by incident_id, service, tool_name. The post-mortem replay tool depends on it; without it, trust collapses on the first incident.

The on-call agent loop is a strict stop_reason FSM. end_turn: success, log + close incident. tool_use: extract the tool call, run hooks, execute, append result, continue. tool_use → hook denied: append the deny as a tool_result with is_error: true, continue (the agent re-plans, usually by escalating). max_tokens: persist partial state, escalate. The loop NEVER branches on response text. The structured stop_reason is the only contract.

Multi-turn ops conversations need the incident metadata anchored. Pin incident_id, service, severity, bucket, partial_status, runbook_in_progress at the top of every system prompt iteration. This survives summarization and ensures the agent on turn 8 still knows it's working on PD-4711, not a generic incident. Without case-facts pinning, multi-turn ops conversations regress to single-turn quality.

Build an eval set of 50 incidents across the 4 buckets + a 'sentiment trap' subset (alerts with angry phrasing but a clean runbook should run the runbook, not escalate). Run the agent over the set; measure: bucket-classification accuracy, correct-runbook rate, false-escalation rate, hook-deny-then-recover rate. Re-run on every change to the runbook registry, hooks, or system prompt.