niklas‑stephan.de

Local context compaction: RustySqueezer fake-LLM compactor (built)

Last updated 05/11/2026 · llm, zed, litellm, compaction

2026-10-31 UPDATE - Zed switched to RustySqueezer for compaction (templates in kb/config). zed-settings.json and zed-settings-macbook.json now declare model rustysqueezer (Metis provider, max_tokens 1000000, tools/images off) and set agent.compaction_model = {"provider":"Metis","model":"rustysqueezer"}. thread_summary_model deliberately stays on Gemini flash-lite (Copilot Haiku on the MacBook): it writes thread titles, which an extractive summarizer cannot do. Verified through the public gateway with a Zed-style compaction prompt on a 27k-token thread: 1,865 tokens back in 0.2 s; the summary keeps task, commands, files, errors and the last turns but NOT old tool-output contents (a fact only present in an early tool result was absent). Zed’s own setting must be edited on each laptop (no settings.json on the servers); Niklas pastes it. Fallback = Gemini flash-lite entry from the old template. Unverified until a real Zed compaction runs: watch docker logs rustysqueezer for [usage] lines.

v1.0.6 (2026-10-30, commit 6bd33a7, pushed): Anthropic native compaction surface implemented and proven end-to-end. POST /v1/messages with a compaction directive in the body (LiteLLM’s auto-router context_compaction child always sends {"compaction":{"type":"summarize"}, "stream": false}) now returns exactly one block {"type":"compaction","content":<summary>,"signature":"rustysqueezer"} with stop_reason: "compaction" - the shape litellm/llms/anthropic/compaction.py parses. Detection is on the body key (Axum’s Json extractor hides headers; the beta header is not required). cargo test 14/14 in CI, eval/surfaces.py gained anthropic-compaction-shape/-parity checks. Verified end-to-end with throwaway containers (ghcr.io/berriai/litellm:main-stable + ghcr.io/handtrixx/rustysqueezer:1.0.6 on a private network, config in /data/docker/ai.handtrixxx.com/test-compaction/config.yaml): a ~15k-token thread through test-router (auto_router + context_compaction model rusty-compactor = anthropic/claude-sonnet-5 → http://rsq-test-106:8000) compacted to a 994-token prompt; containers/network removed afterwards. Docs site (2026-10-30, commits d478347 then 6f69d46, live-verified): the Tailwind/daisyUI single-page site was replaced the same day by a MkDocs + Material docs site (see the Docs site bullet below); routes /, /integrations/{zed,litellm,vscode,opencode}/, /configuration/, /evaluation/, /faq/. README links updated to the new routes. Gotcha: edit_file fuzzy-matching corrupted closing tags in large HTML chunks (</code> doubled); use a python heredoc append + an html.parser tag-balance check before committing.

Open question answered: if the summary + retained tail still exceed the deployment budget, LiteLLM raises 400 ContextWindowExceededError: "The summary and retained conversation still exceed the selected deployment's budget" (observed with max_tokens 1000 vs available 800 = floor(window*trigger_ratio) - max_output). So pick max_tokens comfortably below that available figure. Prod pin done (2026-10-30): RUSTYSQUEEZER_IMAGE bumped 1.0.5 -> 1.0.6 in /data/docker/rustysqueezer/.env, container recreated, --health exit 0, and the compaction surface verified from inside the caddy network (/v1/messages with compaction directive -> stop_reason: compaction). Decision (2026-10-30): Zed keeps its own compaction_model setting (no gateway router needed for Zed); gateway-side compaction is the fallback for clients without such a setting. Still pending Niklas’s go: production litellm_config.yaml entries (rusty-compactor + qwen-router with context_compaction).

  • OpenCode (planned re-host on handtrixxx) - compaction options, verified against official docs + source (opencode.ai/docs/compaction, anomalyco/opencode, 2026-10-30): OpenCode v2 compaction is client-side with a dedicated compaction agent: "agents": {"compaction": {"model": "litellm/rustysqueezer"}} + "compaction": {"auto": true, "keep": {"tokens": 15000}, "buffer": 16000}. Recommended (native): declare LiteLLM as custom provider (npm: "@ai-sdk/openai-compatible", baseURL: https://ai.handtrixxx.com/v1, apiKey via env; models need "limit": {"context": 1000000, "output": ...} since custom providers don’t get limits from models.dev). RustySqueezer already routes as model rustysqueezer in LiteLLM. Caveat: the compaction agent’s summary instruction prompt is OpenCode’s own wording; add its phrasing to RustySqueezer’s instruction_patterns (CONFIG_PATH, roadmap item 1) so it isn’t counted as a user follow-up. Gateway-side (auto-router context_compaction) also works on /v1/chat/completions for any client with no compaction setting (VS Code etc.), but for OpenCode the native agents.compaction.model is cleaner and mirrors Zed. OpenCode’s "compaction": {"type": "native"} per-provider setting is OpenAI-Responses-only, not relevant here. v1.0.4 (2026-10-30): production hardening from the 2026-10-29 readiness audit (roadmap items 8-11): body limit raised from Axum’s 2 MiB default (which 413’d large threads), SIGINT/SIGTERM graceful shutdown (with_graceful_shutdown), per-request usage logging ([usage] <surface> in= out= ratio= latency_ms= on stdout), cargo test in CI on the pinned rust:1.99-alpine toolchain. Prod pin bumped 1.0.2 -> 1.0.4 in /data/docker/rustysqueezer/.env; verified: health ok, eval/eval.py PASS, eval/surfaces.py PASS, graceful drain logged on docker stop.

v1.0.5 (2026-10-30): 1M-token sessions supported and verified. Body limit is now DefaultBodyLimit::max(64 * 1024 * 1024): a 1,004,041-token thread (6.1 MB of JSON) compacts in ~0.1 s inside the container and ~17 s through the LiteLLM gateway (https://ai.handtrixxx.com/v1, model rustysqueezer, needles survive, deterministic); a 72 MB body is rejected with 413. RSS stays ~4 MB even with six concurrent 1M-token requests. Covered by eval/eval.py scenario million-token and unit test million_token_thread_is_compacted (14 tests total). Roadmap item 12 (CPU/memory limits in the deploy compose) was dropped as unnecessary - the compactor is stateless and tiny. Remaining roadmap: 12 (was 13) point Zed’s compaction/summary model at rustysqueezer after quality is accepted on real threads (eval/real_thread.py exists for that).

Current state (PoC, built and verified 2026-10-28)

  • Renamed 2026-10-29 from local-compact to RustySqueezer: repo github.com/handtrixx/RustySqueezer, working copy /data/github/RustySqueezer/, deploy dir /data/docker/rustysqueezer/, container rustysqueezer, image pinned to ghcr.io/handtrixx/rustysqueezer:1.0.2 via env RUSTYSQUEEZER_IMAGE, LiteLLM model rustysqueezer. Old GHCR package local-compact deleted.
  • Hardening + automatic versioning (v1.0.1, 2026-10-29): builder is pinned rust:1.99-alpine (musl target is static by default; crt-static does NOT work with proc-macros - gotcha), runtime is scratch with non-root USER 65534 (~2.9 MB vs 89.7 MB). Healthcheck uses the binary itself: test: ["CMD", "/rustysqueezer", "--health"] (no curl in scratch; --health does a raw TCP GET of its own /health). CI derives the version from Cargo.toml and publishes :latest, :<sha>, :<version>, :v<version> plus a matching git tag v<version> if absent. Dependabot (.github/dependabot.yml) bumps the Docker base image and Cargo.lock weekly; bumps go through CI + eval before merge. Deploy verified: container healthy, eval PASS (189k -> 1284 tokens).
  • v1.0.2 (2026-10-29): Anthropic Messages surface POST /v1/messages (top-level system string, content blocks, SSE streaming, usage as input/output_tokens) and Ollama surface POST /api/chat (NDJSON streaming, response+done, prompt_eval_count/eval_count), POST /api/generate (raw prompt), GET /api/tags. All surfaces return byte-identical summaries; eval/surfaces.py (stdlib, CI step) asserts shape/usage/parity. Docs site docs/index.html rewritten with BeerCSS 5.0.3 Material-Design-3 layout, teal primary (ui("theme", "#00897b") + hardcoded fallback vars). Live on Pages and deployed as 1.0.2 on handtrixxx.com (healthy, eval + surfaces PASS). Working copy /data/github/RustySqueezer has no git remote - update it with git fetch https://github.com/handtrixx/RustySqueezer.git main && git checkout <sha>.
  • Branding/site (2026-10-29): logo rustysqueezer.png (1254x1254) renders under the README headline and in docs/; static GitHub Pages site https://handtrixx.github.io/RustySqueezer/ (Pages enabled via API, source main branch; site is docs/index.html + docs/rustysqueezer.png, root index.html redirects to it).
  • Docs site redesign (2026-10-29): the BeerCSS page looked broken - root cause: in beer.css 5.0.3 .grid is a 12-column grid, so .grid > article children without s*/m*/l* span helpers each get crushed into one 1/12 column; nav.top.max styles the top bar like a sidebar (align-items:flex-start, gap:0). The old page also had <head></head> closing early, dumping all meta/style into the body. docs/index.html rewritten to mirror beercss.com/material-design-3 without a sidebar: plain nav.top bar (logo, title, .max spacer, link buttons, dark-mode toggle), hero = .grid with s12 m6 l6 cells (page bottom active article + logo in page top active), sections section.responsive with h2 + h3.h6 + large-space dividers, cards article.round.no-elevate.large-padding.wave in s12 m6 l4/s12 l6 cells; ui("theme", "#00897b") + ui("mode", "auto") for OS-aware light/dark. Push to GitHub needed for Pages to update.
  • Docs site (current, since 2026-10-30, commit 6f69d46): MkDocs + Material for MkDocs - replaced the Tailwind/daisyUI single-page site (decision by Niklas: he wanted a headroomlabs.ai-style docs site; Material chosen for Markdown-first, sidebar/tabs, built-in search, admonitions, mermaid, dark mode - no frontend build; fumadocs/Docusaurus rejected as React-heavy; note Material for MkDocs is in maintenance mode until 2027-05, successor is Zensical). Structure: mkdocs.yml (nav Home / Integrations Zed+LiteLLM+VS Code+OpenCode / Guide), docs/*.md, docs/stylesheets/extra.css (near-black #05070a + cyan #65d8ff slate scheme, teal light scheme, Inter/JetBrains Mono), docs/javascripts/mermaid.min.js vendored + mermaid-init.js (explicit mermaid.run() on DOMContentLoaded; superfluous CDN-only init renders nothing). Gotchas: upload-pages-artifact defaults to _site (build with mkdocs build -d _site); extra_js is not a key (use extra_javascript); pymdownx.inlinekbd does not exist; ::: grid cards is experimental in material 9.8 - plain <div class="rs-cards" markdown="1"> grid used instead. Pages source switched to build_type: workflow via API (.github/workflows/pages.yml, checkout v7 / setup-python v7 / artifact+deploy v5). Gotcha: switching Pages to workflow while the old legacy build is live made deploy-pages@v3 fail with “Ensure GitHub Pages has been enabled” (404) until the site was re-created via POST /pages; then a push succeeded. Live-verified with Playwright (venv /tmp/pw-venv; chromium needed apt libs libnspr4 libnss3 libasound2t64 etc. on handtrixxx.com).
  • PoC, not production. The service runs and LiteLLM can route to it, but the Zed template (“/data/ai-persistence/config/zed-settings.json) keeps gemini-3.5-flash-liteascompaction_model/thread_summary_model; switching to rustysqueezer` is documented there as a comment only.
  • Source of truth: GitHub repo github.com/handtrixx/RustySqueezer (MIT, public since 2026-10-29; README restructured + CHANGELOG.md added that day, commit 34678a2), working copy /data/github/RustySqueezer/ (push helper push_helper.py removed 2026-10-29; push with GITHUB_PAT from /data/ai-persistence/.env directly). Deploy directory /data/docker/rustysqueezer/ is deploy-only (2026-10-28): compose.yml + .env + README.md, no sources; it pulls the prebuilt image from GHCR. CI (.github/workflows/build-image.yml) has two jobs on every push to main: rust (publish ghcr.io/handtrixx/rustysqueezer:latest and :<sha>) and eval (build the Rust image, run eval/eval.py). The Python PoC (app.py, its Dockerfile/compose, test_thread.py, rust/parity_test.sh and the CI python job) was removed 2026-10-29 - Rust is the only service; Python remains only as the stdlib-only eval harness. The package is public (visibility flipped 2026-10-29), so no login is needed; the host’s stored docker login ghcr.io from the private phase is harmless.
  • Section budgets (since 2026-10-29, commit 1441387): MAX_SUMMARY_CHARS is split per section (section_budgets, percent of the cap, normalised over enabled sections) - tail 50, commands 12, followups 10, key_points 10, errors 8, files 8, original_task 6, tools 4. Headers/titles are subtracted first. Overflow items are dropped with an explicit (+N more ... omitted by the budget) note instead of a hard end-of-summary clip; the tail is filled from the newest turn backwards so the OLDEST tail turns are cut first. [summary truncated] is only a safety net now. Commands clipped to 200 chars, paths to 120.
  • Roadmap (in the repo README): (1) make it generic done 2026-10-28: JSON config file via CONFIG_PATH (schema in config.example.json) covering instruction_patterns, instruction_max_chars, error_pattern, file_extensions, sections (order/enabled) and section_titles (partial merge); env vars still override; defaults reproduce the old Zed behaviour byte-for-byte. (2) rewrite in Rust done. (3) real token counting done 2026-10-28: deterministic word-boundary tokenizer (ASCII alphanumeric/apostrophe runs = 1 token, each other non-whitespace char = 1; identical in Python and Rust, verified equal usage counts; not exact cl100k). (4) eval harness done 2026-10-28, extended 2026-10-29 with a ~190k-token scenario plus ratio and budget assertions; CI now runs it on every push (run python eval/eval.py --url http://<container-ip>:8000/v1 before every deploy). (5) per-section budgets done 2026-10-29 (see Budgeting fix). (6) Anthropic Messages + Ollama surfaces - postponed (2026-10-28). Bug fixed along the way: the commands section kept the OLDEST 25 commands despite the “most recent first” heading; both ports now keep the most recent 25.
  • Deferred (Niklas, 2026-10-28): wiring rustysqueezer into Zed’s compaction, and the Anthropic/Ollama surfaces. The service stays a PoC; Zed keeps Gemini flash-lite.
  • Service: compose project /data/docker/rustysqueezer/ (container rustysqueezer, Rust binary from the GHCR image, internal-only on the caddy network, port 8000 unpublished, no Traefik). Implements /v1/chat/completions (incl. streaming), /v1/models, /health.
  • Algorithm: deterministic extractive compaction - recognizes Zed’s compaction instruction, then emits a structured markdown summary: original task, follow-up user messages, commands run (from tool_calls), files/paths, tools used, error lines, top assistant sentences (TF-IDF-style scoring with recency), last 6 turns verbatim (clipped). MAX_SUMMARY_CHARS (default 12000) is split into per-section budgets, tail first. Verified deterministic (identical input -> identical output).
  • No generative assist (removed 2026-10-29, commit 4ea470d): the ASSIST_* path was measured and dropped. Qwen3.5-0.8B 12-14 s / 2B ~24 s per compaction on this CPU, non-deterministic, and the output is mostly “Thinking Process:” preamble. EmbeddingGemma-2 (--embedding) runs ~100 short sentences/s (~30 s for a 190k thread) and is not a good selector: cosine similarity ranks “Caddy probe 200” vs “Traefik probe 503” at 0.81, nearly as close as paraphrases of the same fact (0.92), so it cannot tell repetition from contradiction. reqwest is gone from Cargo.toml.
  • Key points de-duplicated (2026-10-29): dedup_normalized collapses sentences identical after lowercase/whitespace/punctuation trim before scoring, so repeated tool chatter no longer eats the key_points budget.
  • Gateway (OmniRoute since 2026-10-31; was LiteLLM, where this entry lived in **litellm_config.yaml**): model rustysqueezer = combo -> provider node rsq -> http://rustysqueezer:8000/v1 (verified through the new gateway 2026-10-31; the LiteLLM-specific Auto Router context_compaction and Anthropic-surface notes below apply to LiteLLM only and are historical now). Old detail: max_input_tokens was, max_input_tokens: 1000000. Verified through the gateway: ~26k-token synthetic thread -> ~1.9k-token summary.
  • Zed usage (optional): declare rustysqueezer under the Metis openai_compatible provider with max_tokens: 1000000, then set agent.compaction_model/agent.thread_summary_model to {"provider": "Metis", "model": "rustysqueezer"}. Environment knobs: MODEL_NAME, MAX_SUMMARY_CHARS, TAIL_TURNS/TAIL_CHARS, MAX_COMMANDS/MAX_FILES/MAX_ERRORS/MAX_KEY_POINTS (see .env.example in the repo); per-section shares via section_budgets in the CONFIG_PATH file.
  • Why it beats the 131k limit: the endpoint never feeds the thread to a real model, so over-limit threads are handled fine (see zed-context-compaction-gotchas).

Test command

python3 eval/eval.py --url http://<container-ip>:8000/v1 (the old test_thread.py was removed with the Python PoC). Direct: POST a synthetic thread to http://localhost:8000/v1/chat/completions. Through the gateway: model rustysqueezer with any LiteLLM key.

Deployment state (verified 2026-10-28)

  • Running container: rustysqueezer from ghcr.io/handtrixx/rustysqueezer:latest (Rust build, healthy, internal-only on caddy, port 8000, no Traefik). /health, /v1/models and a synthetic compaction request all verified. Updating = docker compose pull && docker compose up -d in /data/docker/rustysqueezer; pin a build with RUSTYSQUEEZER_IMAGE in .env.
  • Decision (Niklas, 2026-10-28): stay on Gemini until the project is declared stable. “/data/ai-persistence/config/zed-settings.jsonreverted (2026-10-28) tocompaction_model/thread_summary_model={“provider”: “google”, “model”: “gemini-3.5-flash-lite”}; the earlier live switch to rustysqueezer(commit 6027734) was undone. rustysqueezer must NOT be wired into Zed or enabled in LiteLLM use until Niklas says so. Switch procedure when the time comes: deploy the Rust container first, verify health + eval, then addrustysqueezer` (max_tokens 1000000) to the Metis block and point both model keys at it.
  • The gateway (OmniRoute since 2026-10-31) routes rustysqueezer -> http://rustysqueezer:8000/v1 (provider node rsq, see ai-gateway-omniroute; formerly LiteLLM with max_input_tokens 1000000) - no change needed if the Rust container keeps name rustysqueezer and port 8000.

Image hygiene (cleaned up 2026-10-28)

  • Only one local image is kept on handtrixxx.com: ghcr.io/handtrixx/rustysqueezer:latest (89.7 MB, the CI-built Rust binary). The old locally built images rustysqueezer-rustysqueezer:latest (Python PoC) and rustysqueezer-rust:latest were pruned; the :python and pinned-sha tags were also dropped locally; CI no longer publishes :python, and the old :python version was deleted from GHCR on 2026-10-29 (20 versions remain, all Rust).
  • The GHCR package is public since 2026-10-29 - anonymous pulls work. (It started private: the host logged in once with GITHUB_PAT, stored in ~/.docker/config.json; CI uses the built-in GITHUB_TOKEN with packages: write.)
  • Gotcha when checking GHCR visibility: a docker pull on this host proves nothing (credentials are cached). Use an empty config: DOCKER_CONFIG=/tmp/empty docker pull ghcr.io/OWNER/PKG:latest. With curl, the anonymous token works but the manifests are OCI - Accept: application/vnd.oci.image.index.v1+json returns 200, while Docker media types return 404 even for public packages.
  • Gotcha: pushing a locally built image tagged :latest overwrites the CI :latest in GHCR. If that happens, re-point it with docker tag ghcr.io/handtrixx/rustysqueezer:<sha> ghcr.io/handtrixx/rustysqueezer:latest && docker push ... or just rerun the CI workflow. Build images in CI, not on the host.
  • No assist wiring: compose and .env no longer carry ASSIST_* (removed 2026-10-29). The llama-aux containers stay for other aux tasks.
  • Cutover (done 2026-10-28) is just docker compose pull && docker compose up -d; LiteLLM’s api_base is unchanged. To verify a new build before deploying, run it on another port and python3 eval/eval.py --url http://localhost:<port>/v1.

Target (decided 2026-10-28)

Zed’s context compaction (agent.compaction_model, currently gemini-3.5-flash-lite) depends on an external LLM because the local qwen3.8-flashnext (131k ctx) cannot take an over-limit thread as input. Goal: a “fake LLM” — an OpenAI-compatible endpoint that receives Zed’s compaction prompt (the whole thread) and answers programmatically/deterministically with a compact summary (no model call at all - the tiny-model assist was tried and removed after measurement). Register it in litellm_config.yaml as e.g. rustysqueezer (openai-compatible provider) and point Zed’s compaction model at it; no external API needed.

Key insight: Zed’s compaction is just one chat completion whose prompt asks “summarize this conversation so work can continue”. A fake LLM only has to implement /v1/chat/completions + /v1/models and return good extractive text — no tool-calling or agentic behavior required.

Findings (searched 2026-10-28)

No existing product is exactly a fake-LLM compactor for IDE agents, but the pattern exists as transparent context-compression proxies:

  • npow/kompact (github.com/npow/kompact, MIT, pip install kompact, kompact proxy --port 7878): the closest match in spirit. Purely deterministic (no LLM calls): TF-IDF extractive sentence selection, schema/code/log/JSON compressors, observation masker, prefix-cache aligner; sub-millisecond. Anthropic + OpenAI compatible; per-request disable via X-Kompact-Disable. Benchmarked: 40-70% token savings, quality -0.5 to -3.9% on BFCL (vs LLMLingua-2’s -20 to -27% which destroys tool schemas). It compresses requests on the way to the provider; it does not itself act as the summarizing model.
  • 061115xhsm/NeverFull-NeverStop-LLM-Context-Compaction-Proxy (MIT): architecturally identical to Niklas’s idea — transparent OpenAI/Anthropic/Gemini gateway that, when context nears the limit, runs a compaction engine and replaces old messages with a summary; dual-layer deterministic compression (keep 15%, then 50%) plus a pluggable model_hub.py for LLM or local small-model backends, secret redaction, SQLite cache. Quality/unverified (small/unproven repo) — treat as a design blueprint rather than a turnkey tool.
  • headroom-ai/headroom: already deployed here as the compression sidecar on ai.handtrixxx.com (see ai-gateway-omniroute). Deterministic SmartCrusher on older tool/user content, ~33% average savings, prefix-cache friendly. Good building block/precedent, but it compresses ordinary requests, it is not the compaction model itself.
  • microsoft/LLMLingua (EMNLP’23/ACL’24): prompt compression scored by a small model (GPT-2/LLaMA) token-perplexity; LLMLingua-2 is a distilled classifier. Matches the “small local model assists compaction” idea but is heavy and hurts tool schemas; llama.cpp-side KV-cache compaction papers exist but are research-grade.
  • LiteLLM native options (checked against 1.106.2): Auto Router’s compaction only works for OpenAI/Anthropic upstreams (not llama.cpp); litellm.compress() is a beta BM25. Nothing usable for a local model as the compaction target.
  • Hosted/commercial equivalents: Morph Compact (delete-fill compaction API), Anthropic’s server-side “context editing” tool, LangChain deepagents summarization middleware — all either external or LLM-based, i.e. not offline.
  1. Small FastAPI service (OpenAI-compatible /v1/chat/completions, /v1/models, /health) that detects compaction-style prompts and returns a deterministic extractive summary: keep system prompt verbatim, last N turns verbatim, anchor sections (files touched, commands run, decisions, errors) selected by heuristics + TF-IDF/Rank sentences; emit a fixed structure Zed can continue from.
  2. Optionally delegate only the “what remains relevant” judgement to qwen3.5-2b (already running on handtrixxx.com via llama-aux-qwen) — keeps the service fully local.
  3. Register as model rustysqueezer in LiteLLM (api_base = the fake-LLM), set Zed agent.compaction_model to it on all four machines. Must be deterministic/prefix-stable so llama.cpp prefix caching keeps working (Headroom precedent shows this matters).
  4. Validate against the failure mode in zed-context-compaction-gotchas: the fake endpoint must accept threads up to full thread size (it never feeds the whole thread to a model, so no 131k limit problem).

Watch: quality loss of extractive summaries for long agent runs; keep a bypass (e.g. x-headroom-bypass-style header or config switch) to fall back to gemini flash-lite for critical threads.

Budgeting fix (done 2026-10-29, commit 1441387)

The old build assembled all sections and hard-clipped the result at MAX_SUMMARY_CHARS, so on long threads the uncapped “Follow-up user messages” section ate the whole budget and the verbatim tail was dropped entirely - exactly the part Zed needs to continue.

  • Per-section budgets (Config.budgets / config key section_budgets, percent of max_summary_chars, normalised over the enabled sections): tail 50, commands 12, followups 10, key_points 10, errors 8, files 8, original_task 6, tools 4. Headers/titles are subtracted from the budget first. Overflow items are dropped with an explicit (+N more ... omitted by the budget) note; the tail is filled from the newest turn backwards, so the oldest tail turns are dropped first. [summary truncated] is now only a safety net.
  • Commands are clipped to 200 chars, paths to 120.
  • eval/eval.py gained a long-realistic scenario (~189k tokens, 250 tool turns + 60 long follow-ups) with ratio and budget assertions (ratio <= 0.15, summary <= the /health max_summary_chars). It fails on the old image (recall 17 %, tail gone) and passes on the new one (0.7 % kept, 6/6 needles). CI (build-image.yml) now has an eval job that builds the Rust image and runs the harness on every push.
  • Measured: 189552 prompt tokens -> 1233-token summary, deterministic.
  • Eval gotcha: on the handtrixxx.com host :8000 is not rustysqueezer (the container is internal-only; its IP is 172.19.0.20, something else answers on the host port) - use the container IP or --url https://ai.handtrixxx.com/v1 --model rustysqueezer.

Relation to Headroom (checked 2026-10-28)

  • Headroom cannot replace rustysqueezer. Headroom is a request-compression guardrail (already deployed as headroom-proxy:0.41.0 sidecar in front of LiteLLM, see ai-gateway-omniroute); its docs state it “never drops messages” and does no position/score-based context management - it compresses content inside messages (live zone: newest user message + latest tool output) and leaves the history byte-identical. Zed’s agent.compaction_model needs an endpoint that returns a generated summary replacing the whole thread; a compressor cannot serve that role.
  • Complementary instead: Headroom reduces how often compaction triggers (~33 % avg savings on the qwen route); rustysqueezer handles the compaction event itself when the thread exceeds the 131k llama limit.
  • Candidate upgrade (asked 2026-10-28): a Gemma embedding model for semantic turn selection instead of TF-IDF. google/embeddinggemma-2 (GGUF: ggml-org/embeddinggemma-2-GGUF, Q8_0, multimodal via mmproj) is the current model; the homelab already used embeddinggemma (768-dim) for Alexandria vectors before llama-embed was removed 2026-10-27. Caveat: embedding the whole thread on the i7-8700 CPU (~150 tok/s prefill) takes ~15 min for a 150k-token thread - only embed a capped set of candidate sentences/turns, or run it on metis. It cannot replace the extractive summary (embeddings only, no generation).
  • Possible future borrowing: Headroom’s SmartCrusher/CCR ideas (statistical tool-output compression + reversible retrieve-original) could improve rustysqueezer’s extractive stage; Headroom also has an official LiteLLM integration, but the guardrail setup already in place is equivalent.

Production readiness audit (verified 2026-10-29)

What is production-ready: container healthy, internal-only on network caddy (no Traefik route, no published port), image ghcr.io/handtrixx/rustysqueezer public and anonymously pullable, scratch runtime 2.95 MB non-root, self-probing --health healthcheck (exit 0 confirmed), CI builds + runs eval/eval.py and eval/surfaces.py on every push, LiteLLM route rustysqueezer -> http://rustysqueezer:8000/v1 verified end-to-end through ai.handtrixxx.com. Gaps to close before calling it production:

  1. Version drift: /data/docker/rustysqueezer/.env pins :1.0.2; :1.0.3 exists and passes the eval harness (smoke-tested in a throwaway container on port 8001: RESULT: PASS). Bump .env and docker compose up -d.
  2. Not wired into Zed: “/data/ai-persistence/config/zed-settings.jsonstill usesgoogle/gemini-3.5-flash-liteforcompaction_model/thread_summary_model` (intentional until Niklas signs off). RustySqueezer is extractive, so the summary is bullet-fact shaped, not a narrative - test on a real thread before switching.
  3. Axum default 2 MiB body limit -> HTTP 413 Failed to buffer the request body: length limit exceeded. Measured: 1.5 MB OK, 3 MB 413. Real Zed threads are ~1.2 MB (189k-token case), so OK today but thin margin; fix with BodyLimit::max(32*1024*1024).
  4. No request logging/metrics: only a startup println!; no way to see usage volume, input/output tokens or latency.
  5. No graceful shutdown: axum::serve(listener, app).await.unwrap() with no with_graceful_shutdown; docker stop aborts in-flight requests.
  6. Stale local image footgun: on handtrixxx.com ghcr.io/handtrixx/rustysqueezer:latest is the old 88.9 MB Python PoC; compose defaults to :latest when .env is missing, so always pin the version tag.
  7. CI never runs **cargo test** although 9 Rust unit tests exist (config 2, summary 4, text 3).
  8. Minor: CHANGELOG says image ~10 MB, actual 2.95 MB; no compose resource limits.

Tracking (2026-10-29): these gaps are recorded as Roadmap items 8-13 in the repo README (commit 8f870c1, pushed to GitHub). Deliberate decision for this session: no production change yet - prod stays pinned :1.0.2 and Zed keeps Gemini flash-lite for compaction until Niklas signs off.

Release status (2026-10-29)

  • v1.0.3 (2026-10-29, commit 4aadf3a): reliability pass from testing against a real Zed compaction transcript (/home/handtrixxx/code/test.md): tool-result fact lines (versions, IPs, token counts, healthy/PASS/FAIL) now feed the key-point pool (weight 0.6); near-duplicate key points filtered (>=75 % token overlap); sections fit line-by-line instead of stopping at the first overlong line; budgets rebalanced (key_points 10->40 relative share, tail 50->30); error pattern requires real failure context; file regex needs a word boundary (domains like github.com no longer match as github.c). New harness eval/real_thread.py (stdlib): parses ## User/## Assistant transcripts into Zed’s exact message shape incl. tool_calls+tool results, checks needle recall, determinism, budget, tail, surface parity. Real thread: 22061 -> 2898 tokens (13.1 %), 23/25 needles, determinism/budget/tail/parity PASS -> RESULT: PASS. CI publishes :1.0.3/:v1.0.3 tags automatically from Cargo.toml.
  • v1.0.0 tagged with a GitHub release (github.com/handtrixx/RustySqueezer/releases/tag/v1.0.0); crate version 1.0.0. CI rebuilds :latest and the pinned sha on each push.
  • Open items closed 2026-10-29: GHCR package is public (toggle under “Package settings”, there is no “Manage” button for user-namespace packages), social preview image set in the web UI. Only remaining gate for real use: Niklas’s go-ahead to point Zed’s compaction_model/thread_summary_model at RustySqueezer.

VS Code / Copilot compaction (checked 2026-10-30)

  • No compaction-model override exists. VS Code’s automatic//compact summarization is performed by the model selected for the session, not by chat.utilityModel. Verified in microsoft/vscode source (extensions/copilot/src/extension/intents/node/agentIntent.ts, handleSummarizeCommand -> endpointProvider.getChatEndpoint(request)); the only compaction settings are github.copilot.chat.summarizeAgentConversationHistory.enabled, ...Threshold and ...agentHistorySummarizationMode.
  • Consequence: RustySqueezer (extractive, no tool calling) cannot be a drop-in compaction model in VS Code the way it is in Zed. A BYOK model needs toolCalling: true to appear in the agent model picker.
  • Path that does work: point VS Code’s Custom Endpoint provider (vendor: "customendpoint", apiType: "chat-completions") at a RustySqueezer passthrough/proxy mode that compresses the incoming prompt and forwards the real response to LiteLLM, with contextWindow declared large so VS Code never needs its own compaction. Same shape as the Headroom sidecar in the gateway. Not implemented yet.
  • Experimental alternative: apiType: "responses" lets VS Code send a server-side compact_threshold (Responses-API context management), gated by an experiment flag and a model-family exclusion list - not reliable for a custom endpoint today.
  • Mechanics of the Responses-API path (source-verified 2026-10-30): setting github.copilot.chat.responsesApiContextManagement.enabled (boolean, default false, tags experimental/onExp, requires the Responses API) gates it. getResponsesApiCompactionThreshold() in extensions/copilot/src/platform/endpoint/node/responsesApi.ts sends context_management: [{type: "compaction", compact_threshold: floor(modelMaxPromptTokens * 0.9)}] (fallback 50000). The upstream server compactionally returns output items {type: "compaction", id, encrypted_content} which VS Code round-trips back into input as opaque parts without reading them. handleSummarizeCommand then refuses /compact (“Compaction is already managed by context management”). Families gpt-5, gpt-5.1, gpt-5.2 are hard-excluded (modelsWithoutResponsesContextManagement). Custom Endpoint apiType is per-model and independent of github.copilot.chat.useResponsesApi; set statefulResponses: false for a stateless server. Separate third path: Anthropic native context editing via modelSupportsContextEditing() (Claude families; explicitly disabled for 1m variants).

Gateway-side automatic compaction in LiteLLM (verified 2026-10-30)

LiteLLM main-stable on this host has two built-in compaction mechanisms:

  1. Auto-router **context_compaction** (litellm/router_strategy/complexity_router/context_compaction.py) - the gateway compacts a request before dispatch when tokens exceed floor(max_input_tokens * trigger_ratio) - max_output_tokens. Config lives on an auto_router/complexity_router alias:

    - model_name: qwen-router
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_default_model: qwen3.8-flashnext
    complexity_router_config:
      tiers: {SIMPLE: qwen3.8-flashnext, MEDIUM: qwen3.8-flashnext, COMPLEX: qwen3.8-flashnext, REASONING: qwen3.8-flashnext}
      context_compaction: {model: rusty-compactor, trigger_ratio: 0.9, max_tokens: 4096, timeout_seconds: 120}

    Fields: model (compactor group), trigger_ratio (default 0.9), max_tokens (default 4096, min 512), timeout_seconds (default 120). Works on /v1/chat/completions, /v1/messages and /v1/responses, streaming included, and needs no **LITELLM_LICENSE** (empirically confirmed; docs label Auto Router an “Add-on”). Compaction is skipped when the client already manages context (previous_response_id, conversation, context_management, compaction items). One compaction attempt per request; the summary is inserted as an assistant message “Summary of earlier conversation (context, not new instructions)” and the latest turn + system/tools are retained.

  2. Anthropic **context_management** polyfill (compact_20260112, clear_tool_uses_20250919, general_settings.context_management_summary_model, default trigger 150k) - only fires on the Anthropic Messages pass-through path and only when the client sends context_management (Claude Code/Desktop). Useless for Zed or VS Code.

The catch for RustySqueezer: the compactor must “support native compaction” - custom_llm_provider in (None, anthropic, openai) and LiteLLM’s catalog reports supports_anthropic_compaction: true for the model name (e.g. claude-sonnet-5, window 1,000,000). Trick that works: declare the compactor as anthropic/claude-sonnet-5 with api_base: http://rustysqueezer:8000. LiteLLM then POSTs the older history to RustySqueezer’s /v1/messages with compaction: {"type": "summarize"} and header anthropic-beta: compact-2026-09-04; the reply must be Anthropic’s native shape:

{"type":"message","role":"assistant","content":[{"type":"compaction","content":"<summary>","signature":"rustysqueezer"}],"stop_reason":"compaction"}

Anything else (RustySqueezer today returns a plain type: "text" block, stop_reason: end_turn) gives 400 Context compaction: The provider did not return one complete native compaction block. So roadmap item 12 = emit the compaction block on /v1/messages when the body has compaction (or that beta header). Not implemented; needs Niklas’s sign-off.

  • Proof of mechanism: throwaway container litellm-compact-test (port 4001, config + mock compactor in /data/docker/ai.handtrixxx.com/test-compaction/) compacted a ~2.4k-token thread to 48 prompt tokens on all three surfaces, streaming included. Container removed after the test; the config/mock remain as reference.

Plan (decided 2026-10-30, not yet executed)

Goal: gateway-side automatic compaction so long Zed (and later VS Code) sessions never die on the 131k window, using RustySqueezer as the compactor instead of a cloud summary model.

  1. RustySqueezer v1.0.6 (repo /data/github/RustySqueezer, ~30 lines in rust/src/main.rs anthropic_messages): when the request body contains compaction (or header anthropic-beta: compact-2026-09-04), return the summary as {"type":"compaction","content":...,"signature":"rustysqueezer"} with "stop_reason":"compaction"; otherwise behave exactly as today. Add a compaction scenario to eval/eval.py/eval/surfaces.py.
  2. Gateway wiring in /data/docker/ai.handtrixxx.com/litellm_config.yaml: add rusty-compactor as anthropic/claude-sonnet-5 -> api_base: http://rustysqueezer:8000, and a qwen-router alias (auto_router/complexity_router, all tiers = qwen3.8-flashnext, context_compaction: {model: rusty-compactor, trigger_ratio: 0.9, max_tokens: 4096}). Zed then points at qwen-router with a large declared max_input_tokens and no cloud compaction_model. Needs a LiteLLM restart - ask Niklas before doing it.
  3. VS Code: no Option-A passthrough proxy (rejected). Declare the custom-endpoint model’s contextWindow as 1M so Copilot never self-compacts, and route it through the same qwen-router; the gateway compacts transparently. Open question for next session: whether LiteLLM errors (400) when summary+tail still exceed the budget - tune max_tokens/trigger if so.
  4. Zed side stays Gemini flash-lite until RustySqueezer output quality is signed off on real threads (roadmap item 7 + the harness eval/real_thread.py).

Source: Wiki.js