Zed Agent + Local Model: Context Compaction Gotchas

Last updated 05/11/2026 · migrated-from-outline, llm, zed

Diagnosed 2026-10-04 against the then-current setup (Open WebUI at ai.handtrixxx.com/api → 27B; Open WebUI is retired and prod is llama-flashnext now - see llm-serving-metis). The mechanism below is still valid. Symptom: auto-compaction “runs forever and never finishes,” manual /compact also hangs, and eventually the hard error:

"detail": "request (192213 tokens) exceeds the available context size (131072 tokens), try increasing it"

Root cause (the part that’s easy to get wrong)

Compaction re-sends the ENTIRE thread history as a single prompt to the LLM (by default the thread’s current model; configurable via agent.compaction_model). So /compact = “here are all N messages, summarize them,” and that one summarization request must itself fit in the context window.

Consequence: once raw history > the window, compaction is mathematically impossible. The summarization prompt is bigger than 131k, so llama.cpp rejects it before any summary exists. Zed has no graceful error surface for this against a local/OpenAI-compatible provider → it looks like it “hangs forever.” No **settings.json** change can recover a thread that’s already over the limit. You must start a fresh thread (“New From Summary” from the +/New Thread menu, or just a clean thread — real state is in the files, not the chat).

Why auto-compact didn’t stop it (two ways to overshoot)

Zed config had "auto_compact": { "enabled": true, "threshold": "80%" } → fires at ~104,857 tokens on a 131k window. Yet the thread reached 192k. Two mechanisms:

A — single-turn spike: auto-compact runs on message boundaries. One giant tool result or @-mention (big file read, huge terminal output) can jump the thread from under ~105k to over 131k in a single step, before any compaction check runs. ~87k tokens = one large file dump — very plausible in agentic work.
B — counter mismatch: Zed counts tokens client-side with its own tokenizer; llama.cpp counts with the real Qwen tokenizer. For a local model behind Open WebUI these frequently disagree, and Zed’s estimate tends to run low. So Zed thinks you’re at 100k (below trigger → no compact) while llama.cpp already sees 135k+. The token counter near the model selector is slightly lying; that gap is where threads slip through.

The fix

1. Use an absolute “tokens remaining” threshold, not a percentage. This neutralizes both A and B:

"agent": {
  "auto_compact": {
    "enabled": true,
    "threshold": -30000   // compact once < 30k tokens remain → fires at ~101072 used, anchored to the live 131072 ceiling
  }
}

Live on the ThinkPad and in the template “/data/ai-persistence/config/zed-settings.json (-30000; history: 80%→-30000(2026-10-19, vs 196608) →-25000(2026-10-21, vs 131072) →-45000briefly on 2026-10-22, **reverted the same day by Niklas** because firing at \~86k leaves too little context to actually work in). Why-Nbeats80%: with a percentage, a single big turn leaps straight over the trigger. -Nanchored to the known ceiling is robust to the Zed-vs-Qwen counter undershoot. At-30000` the peak after one normal turn is ~111k; only a giant single tool call can still break through, which is what item 2 prevents.

2. Chunk large file writes (root cause of the 2026-10-22 Alexandria case). Writing a whole source file (e.g. a Rust module in the Alexandria/memory-mcp project) in one write_file call generates 10-20k tokens inside a single turn, entirely between compaction checks — no threshold can catch that. Rule now in agents.md (rendered into every server’s AGENTS.md): keep write_file to ~200-300 lines, append the rest with successive edit_file calls, avoid @-mentioning big files, and bound terminal output with head_lines/tail_lines.

2. **agent.compaction_model** on a big-window cloud model (DECIDED 2026-10-22). Docs: “The configured model should have a context window at least as large as the thread’s primary model.” Default = the thread’s own model (same 131k ceiling → compaction is mathematically impossible once the thread is over it). Set it to a Gemini model with a 1M window, which makes compaction never fail:

"language_models": {
  "google": {
    "available_models": [
      { "name": "gemini-3.5-flash-lite", "display_name": "Gemini 3.5 Flash Lite (1M ctx)", "max_tokens": 1000000 }
    ]
  }
},
"agent": {
  "compaction_model":      { "provider": "google", "model": "gemini-3.5-flash-lite" },
  "thread_summary_model":  { "provider": "google", "model": "gemini-3.5-flash-lite" }
}

Custom entries are keyed by model name and override that model’s declared attributes (docs, use-api-access.md) - they do not replace the provider’s whole catalog.
thread_summary_model matters too: “New From Summary” is the other escape hatch and needs the same big window.
One request per compaction event, not per turn: auto-compact fires at ~101k, the thread shrinks to ~10-20k, so each compaction buys ~80-90k of further work - roughly 3-6 calls in a long Alexandria session. Prefill of 80-120k input tokens on flash-lite is seconds, much faster than a cold prefill on metis.
Limits: Gemini free tier is per project, measured in RPM and TPM/TPD, and Google no longer publishes fixed free numbers for text models - check the model’s limits in AI Studio. A few compactions per hour is fine; repeated manual /compact in one minute can trip TPM (429 RESOURCE_EXHAUSTED → wait and retry).
Caveats: every compaction ships the whole thread (source, terminal output, KB/AGENTS.md host topology) to Google - same exposure as flash-lite already being the default model. If summaries feel lossy, switch to flash (still 1M). Compaction also drops the llama prompt-cache entry, so the next primary turn is a cold prefill (that cost exists today too).
NOT a license to run big threads: the primary model still rejects anything above its 131072 window, so the threshold + chunked writes still have to keep the thread under it.

3. Habit: “New From Summary” between major tasks. At 131k in an agentic workflow that dumps big tool results, hitting the wall is normal. Seeding a fresh thread from a summary keeps each thread comfortably under the limit so compaction always has room to work.

Can we host a bigger compaction model locally? (re-evaluated 2026-10-26: still no, and the reason is compute, not RAM)

The binding constraint is prefill of the whole thread, not weights. Compaction is one request containing the entire history, so throughput is prompt-processing (compute-bound, scales with cores/GPU), while KV is only linear memory. A box needs BOTH a window >= thread size and >=100 tok/s prefill.

Prefill of a ~131k prompt by host (4B-class model, estimates from measured rates):

metis (Strix Halo iGPU): real cold prefill 200-370 tok/s -> ~6-11 min. The only box in range.
handtrixxx.com (i7-8700, 12c, AVX2 only): ~15-20 tok/s -> ~2 h. RAM is not the problem (64 GB, ~13 GiB free, 57 containers); compute is.
hestia (Pi 5): measured 16.6 tok/s with Qwen3-4B Q4_K_M (2026-10-26), docs 12-14 tok/s -> ~2-3 h.
iris (Pentium G4400, 2c): ~5-8 tok/s -> 4+ h. Explicitly “avoid heavy workloads”.
ThinkPad: same CPU-prefill problem, plus it would have to be configured per laptop.

KV cache is the other trap: small dense models are NOT cheap at long context. Per token (f16): dense Qwen3-4B/Llama-3.1-8B class 128 KiB -> 25 GB at 196k; Qwen3.5-4B (hybrid, Gated DeltaNet, only 1/4 of layers full attention) 32 KiB -> 6.3 GB f16, ~3.4 GB q8_0, ~1.8 GB q4_0. MoE does not help KV at all. So a local compaction model must be a hybrid (Qwen3.5/3.8 family, Nemotron Lightning, Gemma sliding-window) or KV blows the budget.

Verdict per host (2026-10-22 numbers still hold: metis GTT 44.3/56 GiB, ~14.2 GiB RAM free; a second resident llama-server adds its own weights + several GiB of KV and starves the 28.8 GB weight shard’s page cache).

The realistic local route is not a second model on a second box - it is raising the window on the model metis already runs: -c 196608 on llama-flashnext (measured 46.0 GiB GTT idle, ~4.6 GiB RAM available - the documented fallback), keeping Zed’s declared max_tokens at 131072 so the primary turn stays inside the window while the compaction request (thread + instructions, ~140k) still fits physically. Two slots in one server do NOT work: -c is the total shared across slots, so -np 2 halves each request’s window.

Unverified idea worth a bench: llama.cpp prompt caching could make local compaction nearly free, because the compaction prompt shares the thread’s prefix - only if the variable instruction text goes last and --cache-ram (8192 today) is large enough to hold the thread’s KV (~2.2 GB for a hybrid at 131k). Chunked/recursive summarization is the only way a CPU box could do it without a big window, but total prefilled tokens are the same or higher, so it is still metis-only.

To confirm which mechanism you hit (next time)

Before abandoning the thread, look at the token counter near the model selector when compaction first struggles:

Shows ~130k+ but Zed never compacted → B (counter undershoot / one big jump). Fix: keep -30000 + the cloud compaction_model; lower the threshold only if it keeps happening.
A single message ballooned the count by tens of thousands → A. Same fix; also avoid @-mentioning huge files or dumping giant terminal output (use head/tail, target reads).

What is NOT the problem (verified)

llama-server reports the real window via /v1/models (n_ctx vs n_ctx_train) - the Zed-side max_tokens must match the live -c (currently 131072 for llama-flashnext since 2026-10-21; was 196608).
Zed’s per-model max_tokens must match llama’s real -c (131072 then; currently provider key Metis in zed-agent-direct-llama-setup). Provider wiring is fine - the bug is purely the compaction mechanics + counter, not a misconfigured window.

Raising the context (evaluated 27B-era - do NOT use compaction as a reason to raise it)

Numbers are from the 27B era; prod runs Flash-Next at -c 131072 (196608 until 2026-10-21). The general lesson stands: more context = more KV = slower decode.

Tempting to bump llama -c 131072 -> -c 262144 (n_ctx_train is already 262144, so supported). Rejected as the fix:

KV is a Gated DeltaNet hybrid (only 16/64 layers hold full attention KV) → 128k ≈ 4.3 GiB; 256k ≈ +8.5 GiB more KV.
Only ~2 GiB free in the VRAM carve-out (29.9/32 used), so the extra KV spills into GTT → ~3.8 + 8.5 ≈ ~12.3 GiB GTT. RAM can technically hold it, but:
- GTT-resident KV is slower for decode than VRAM-resident (see “Local LLM Serving on Metis”). Doubling GTT-KV likely drops the ~30 tok/s decode meaningfully.
- More DRAM bandwidth contention (the same concern documented for gaming alongside the LLMs).
Net: trading decode speed to avoid compaction isn’t worth it. The compaction ceiling is solved off-box (Gemini 1M) instead of by raising -c; -c 196608 (192k, ~+4.2 GiB KV) remains the fallback if local headroom is ever genuinely needed.

Copilot Chat models: context window is NOT overridable (learned 2026-10-26)

agent.compaction_model accepts only {provider, model}. Unlike openai_compatible/google, **copilot_chat** has no **language_models** settings section and no way to declare **max_tokens** - Zed takes the window straight from GitHub’s Copilot model catalog. Source (zed-industries/zed, main): crates/copilot_chat/src/copilot_chat.rs parses capabilities.limits.max_context_window_tokens / max_prompt_tokens; crates/copilot_chat/src/model.rs sets max_input_tokens = max_prompt_tokens().unwrap_or(max_token_count()). So whatever Copilot’s /models endpoint reports is what Zed trusts - a 256k display is GitHub’s catalog value, not a Zed cap.

Claude Haiku 5.5 went GA in Copilot 2026-10-07 with a 1M window (github.blog changelog); on Business/Enterprise the org admin must enable it in the Copilot model policy. Zed reads the same catalog VS Code does, so the fix for “want Haiku 5.5 (1M) as compaction model” is: update Zed, find the exact Haiku 5.5 entry in the model picker (expected id claude-haiku-5-5) and use that id in compaction_model/thread_summary_model - a config pinned to claude-haiku-4.5 gets the 4.5 entry (~200k) instead. Could not verify the bbraun.ghe.com catalog directly: GITHUB_PAT in /data/ai-persistence/.env only authenticates on github.com (401 on api.bbraun.ghe.com).

If the org catalog genuinely reports 256k for Haiku 5.5 on the chat endpoint, there is no in-Zed override; the only self-hosted escape is routing compaction through LiteLLM’s (unofficial) github_copilot provider on ai.handtrixxx.com and declaring max_tokens: 1000000 via an openai_compatible provider block.

Note: a 256k compaction model is fine whenever the thread’s primary model window is <= ~230k (e.g. llama-flashnext at 131k). The 1M requirement only matters for threads whose primary model has a bigger window.

Zed settings keys involved

agent.auto_compact.enabled (bool), agent.auto_compact.threshold ("80%" | positive int = tokens used | negative int = tokens remaining; 0 invalid → falls back to 90%).
agent.compaction_model ({provider, model}) — which model does /compact + auto-compact. Default = thread’s current model. Set to google/gemini-3.5-flash-lite on all machines (template “/data/ai-persistence/config/zed-settings.json`, ThinkPad live since 2026-10-22).
agent.thread_summary_model — the model behind “New From Summary”; give it the same big window.
Per-model max_tokens under language_models.openai_compatible.<provider>.available_models[] — the context window Zed trusts for that model.