- ✓Route Claude Code through Llama.cpp to a local Gemma 4 31B model and you get 22-28 tokens/sec autonomous coding, zero API spend, and data that never leaves your NVMe. The trick is two env vars (ANTHROPIC_BASE_URL, ANTHROPIC_API_KEY) plus the free-claude-code bridge, a 32768-token context window, and a one-line CLAUDE_CODE_MAX_TOKENS=16384 ceiling so the client doesn't try a 200k context and wipe the run.
- ✓The local stack runs the same Claude Code client - every Claude Code architecture question still applies, the inference layer just moves.
- ✓Local inference is the highest-leverage privacy + cost lever for personal-productivity agents.
- ✓Anthropic-format tooling is smart, not psychic - the repo-mapping CLAUDE.md tip in this post is a direct application of the hierarchy concept.
Quick answer
The Local Bridge Stack = Claude Code client + ANTHROPIC_BASE_URL redirect + free-claude-code proxy + Llama.cpp + Gemma 4 31B. Output: 22-28 t/s autonomous coding on an RTX 4090, zero API bill, data stays local. Two non-obvious env vars do the heavy lifting (CLAUDE_CODE_MAX_TOKENS=16384, --api-type anthropic). A root CLAUDE.md mapping the repo architecture is the highest-leverage configuration step after the env vars.
The setup, in 11 lines
Running Anthropic Claude Code with Llama.cpp and Gemma 4 has a new sweet spot. Before: weekend science project. After: a stack worth running every day.
- Route Claude Code to local inference with
ANTHROPIC_BASE_URL="http://localhost:8080/v1"so Anthropic-format calls hit Llama.cpp. - Set
ANTHROPIC_API_KEY="sk-local-token"anyway. The bridge accepts any placeholder; the client refuses to start without one. Yes, it feels illegal. - Install the modern stack with
uv, not pip detours:uv tool install @anthropic-ai/claude-code. - Don't use old
uv. Pre-v0.6.0 reportedly breaks free-claude-code's proxy dependency resolution. Upgrade first. - Run the bridge from
free-claude-codeso Claude Code can talk to a local server without pretending it's a cloud app. - Start Llama.cpp with
--ctx-size 32768. Smaller contexts look fine until repo analysis lands and everything catches fire. - Add
--cont-batching- Claude Code isn't one clean completion. It's a think/act loop, and batching matters. - On newer Llama.cpp builds, use
--api-type anthropicso/v1/messagesworks natively. Less glue code, fewer haunted bugs. - Set
CLAUDE_CODE_MAX_TOKENS=16384. Most people miss this. If you don't set it, Claude Code may try a 200k context and wipe your local run. - Pick the right Gemma variant.
Gemma 4 31B Q4_K_Mis the practical logic/speed tradeoff; the26B-A4B MoEis faster but loops more on harder refactors. - The payoff: Gemma 4 31B on an RTX 4090 reportedly runs autonomous coding at 22-28 tokens/sec, with data staying local instead of leaving your NVMe.
The bonus tip nobody mentions
Create a root CLAUDE.md mapping the repo architecture. Top-level dirs, module roles, where side-effects live, entry points. Anthropic-format tooling is smart, not psychic - and a structural index in CLAUDE.md gives the local model navigational grip even when individual file contents aren't in context. Twenty minutes of authoring; pays back across every session.
Where it still goes weird
Three honest failure modes:
- Deletions over 500 lines get weird. The model's reasoning about cross-file impact thins out at that scale.
- Repo-wide refactors need explicit chunking via Plan Mode. Don't ask for them in one prompt.
- The 32768 context is enough for most files but not whole-repo analysis. Pair with the root
CLAUDE.mdabove.
This isn't perfect. It's just good enough to be the default for most personal-productivity coding work.
How this shows up on the exam
D3 (Claude Code Configuration) repeatedly tests whether you understand that the client and the inference layer are separable concerns. Most candidates assume the Claude Code workflows (Plan Mode, CLAUDE.md hierarchy, Skills, hooks, slash commands) are tied to Anthropic's API. They're not - the protocols are public, the bridge layer is real, and the workflows survive a model swap. Exam questions that ask "where does configuration X live?" expect you to know it lives in the client and the file system, not in the API.
D5 (Context Management) tests the same pattern from the other side. The CLAUDE_CODE_MAX_TOKENS=16384 ceiling and --ctx-size 32768 are the worked example of deliberate context budgeting - you don't just take whatever the default is, you size the budget to the model and the task. The exam will distract you with "raise the context limit" answers; the correct architecture is to bound the budget and use Plan Mode + CLAUDE.md to make the bounded context productive.
If you're running a different local stack, what token/sec are you seeing - or where's the bottleneck?
That's the question worth asking around. The local-Claude-Code field is moving fast enough that the sweet spot in May 2026 won't be the sweet spot in August. But the architectural patterns - env-var redirect, bridge proxy, deliberate context budgeting, structural CLAUDE.md - survive every model swap.
4 direct routes into the scored material.
- scenariosThe local stack runs the same Claude Code client - every Claude Code architecture question still applies, the inference layer just moves.
- scenariosLocal inference is the highest-leverage privacy + cost lever for personal-productivity agents.
- conceptsAnthropic-format tooling is smart, not psychic - the repo-mapping CLAUDE.md tip in this post is a direct application of the hierarchy concept.
- conceptsThe 32768 ctx-size + CLAUDE_CODE_MAX_TOKENS=16384 ceiling is a worked example of why context-window discipline matters in production.
Share this post.
One share is one less person stuck on the same question.