niklas‑stephan.de

AI gateway: agentgateway (ai.handtrixxx.com + mcp.handtrixxx.com) - replaced OmniRoute 2026-11-01

Last updated 06/11/2026 · agentgateway, omniroute, gateway, mcp, handtrixxx, llm

Single Rust data plane for LLM (https://ai.handtrixxx.com/v1) and MCP federation (https://mcp.handtrixxx.com/mcp and /sse). Replaced OmniRoute on 2026-11-01 (decision by Niklas, see mcp-gateway-options follow-up). The Headroom compression sidecar was removed in the same step (Niklas: weird implementation, a better compression approach may come later; the evaluation results are in the archived ai-gateway-omniroute).

Layout (project /data/docker/ai.handtrixxx.com/, flat)

  • compose.yml: one container agentgateway (cr.agentgateway.dev/agentgateway:v1.7.0, network caddy, command: -f /config/config.yaml). config.yaml is mounted read-only; SQLite state in ./data (owned by uid 65532, so moving it needs sudo).
  • Ports: only 127.0.0.1:8086 (data plane, local smoke tests). The admin UI is not published on a host port; config.adminAddr: 0.0.0.0:15000 makes it reachable inside the caddy network for Traefik.
  • Traefik: router agentgateway-api = ai.handtrixxx.com + /v1 (priority 20); agentgateway-mcp = mcp.handtrixxx.com; admin UI = https://agentgateway.handtrixxx.com/ui/ (router agentgateway-admin -> service port 15000, middleware authentik-agentgateway forward-auth) plus agentgateway-outpost (/outpost.goauthentik.io/, priority 15). Internet-reachable but behind authentik (proxy provider 47 “Provider for agentgateway” / app agentgateway, forward_single, attached to the Embedded Outpost, policy-bound to user handtrixxx). Unauthenticated -> 302 to auth.handtrixxx.com. https://ai.handtrixxx.com/ itself falls through to a 301 to handtrixxx.com. The admin UI has no auth of its own: never publish port 15000 on the host.
  • .env (mode 600, names only): GEMINI_API_KEY, SEMANTIC_MCP_TOKEN, HASS_TOKEN, GITHUB_MCP_TOKEN, CONTEXT7_API_KEY, SESSION_KEY (six; OUTLINE_KEY removed 2026-11-06 with the outline target). Config change = edit config.yaml, then docker compose up -d (restart). SESSION_KEY (random openssl rand -hex 32) is wired via config.session.key: $SESSION_KEY and encrypts MCP session IDs (AES-256-GCM); without it the read-only config cannot persist a generated key and the gateway warns. Do not lose or rotate it casually: sessions break until clients re-initialize (harmless, they do that automatically).
  • Smoke tests in the project dir: python3 smoke.py (chat, stream, tool call, bad key against 127.0.0.1:8086) and python3 mcp_smoke.py (initialize + tools/list, expects 81 federated tools).

Keys

  • gateways.default.apiKey.mode: strict with keyHash: sha256:<hex> entries in config.yaml. The hashes are of the existing OMNIROUTE_KEY_* values in /data/ai-persistence/.env (ZED_MACBOOK, ZED_THINKPAD, ZED_METIS, ZED_IRIS, ALEXANDRIA, AGENT), so every client kept its key string and URL; nothing had to change client-side. The .env variables were renamed OMNIROUTE_KEY_* -> GATEWAY_KEY_* on 2026-11-02 (values unchanged); the old names no longer exist anywhere.
  • Each key carries metadata: { group: ... } (2026-11-02), readable in CEL policies as apiKey.group: the 6 laptop keys (ZED_MACBOOK, _ZED_THINKPAD, _ZED_METIS, _ZED_IRIS, _ZED_HANDTRIXXX, ALEXANDRIA) are group: laptop, the AGENT key (OpenChamber) is group: agent. ZED_HANDTRIXXX was added 2026-11-02 for the Zed sessions in the /data workspace on handtrixxx.com itself. This is what the per-client MCP visibility below branches on.
  • Each key has allowedModels = the 8 public model names. New key: generate a random value, store it in /data/ai-persistence/.env, add sha256: of it plus allowedModels to config.yaml, restart.

Models

Public names unchanged: qwen3.8-flashnext, qwen3.5-2b, gemma-4-e2b-mtp, rustysqueezer, gemini-3.5-flash, gemini-3.5-flash-lite, gemini-3.1-pro (= gemini-3.1-pro-preview), gemini-flash-latest.

  • Internal custom providers (visibility: internal, OpenAI completions format): metis/* -> http://100.64.0.1:8000/v1, auxqwen/*, auxgemma/* -> llama-aux containers, rsq/* -> http://rustysqueezer:8000/v1. Gemini = native provider: gemini with $GEMINI_API_KEY.
  • virtualModels (OmniRoute combos’ replacement) use routing.failover.targets with priorities. qwen3.8-flashnext now goes directly to metis (no Headroom).
  • No compression anywhere on the gateway path. Gemini free/paid quota errors (429/503) are upstream, not a gateway fault, so do not use Gemini calls as a health check.

MCP federation (mcp.targets, failureMode: failOpen, prefixMode: conditional)

  • semantic -> https://docs.niklas-stephan.de/semantic/mcp (SEMANTIC_MCP_TOKEN, 3), homeassistant -> http://100.64.0.3:8123/api/mcp (HASS_TOKEN, 28), context7 -> https://mcp.context7.com/mcp (CONTEXT7_API_KEY, 2 tools; key lives in /data/ai-persistence/.env on metis and was copied into the gateway project .env on handtrixxx.com; Context7 validates it on tool calls but not on initialize), github -> https://api.githubcopilot.com/mcp/readonly (GITHUB_MCP_TOKEN in project .env, a copy of GITHUB_PAT; 28 read-only tools). 78 tools in total as of 2026-11-06 (outline’s 20 dropped); tool names are prefixed with the target name (wiki_*, github_*, context7_*).
  • HA’s /api/mcp is already the aggregate of HA’s individual endpoints (verified 2026-11-01 by comparing tools/list: 26 + 1 + 1 = 28 tools, each prefixed with its API name: assist__* = /api/mcp/assist, home-assistant-services__HassCallService = /api/mcp/home-llm-service-api, frigate-services__frigate_query = /api/mcp/frigate_services). Niklas decided to keep only the aggregate; adding the individual endpoints would duplicate tools.
  • GitHub is read-only on purpose: GITHUB_PAT is a broad classic token (delete_repo, admin:org/enterprise, workflow…), so any gateway key holder would otherwise get those powers. Switch to write by changing the host to https://api.githubcopilot.com/mcp/ (toolsets via X-MCP-Toolsets header or /x/<toolset> path), ideally after replacing the PAT with a fine-grained token.
  • Per-client tool visibility (2026-11-02): the github target carries condition: 'apiKey.group == "agent"', so laptop keys see outline + semantic + homeassistant + context7 = 53 tools ≈ 12.1k tokens; agent keys see all 81 tools ≈ 31.3k tokens. homeassistant is deliberately visible to everyone. Measured 2026-11-02 via tools/list: all 81 tools = 125,062 chars ≈ 31.3k tokens (github ~19.1k, outline ~5.9k, homeassistant ~4.4k, context7 ~1.3k, semantic ~0.5k). A fresh Zed session prefills its visible list cold (new slot = no cache); with the laptop set that is now ~12k instead of ~25-31k. To change visibility: edit the target condition / key metadata.group in config.yaml, restart.
  • Clients use the same gateway keys as for the LLM API.

Verified 2026-11-01 after cutover

All 6 client keys 200 on the public URL; qwen3.8-flashnext, qwen3.5-2b, gemma-4-e2b-mtp, rustysqueezer 200; bad key 401; mcp.handtrixxx.com/mcp without key 401; local smoke + MCP smoke pass. Gemini was seen working in the trial (200) but not re-tested publicly (rate limits).

Cleanup done 2026-11-01 (no rollback to OmniRoute anymore)

  • Deleted: omniroute-retired/ (compose, data, headroom build, tests), images diegosouzapw/omniroute:3.8.52 and headroom-proxy:0.41.0 (plus dangling images, 5.3 GB), authentik app/provider omniroute (provider 46), /data/maintenance/omniroute-oidc/. OMNIROUTE_KEY_* names in /data/ai-persistence/.env stay (they are the live client keys).
  • litellm-retired/, the LiteLLM Postgres db/, the authentik app litellm (provider 40) and the /tmp/*.bak-* working backups were all deleted (same day, Niklas approved). Nothing LiteLLM-related remains in the project.
  • Gotcha (authentik): a proxy provider created via manage.py shell (ORM) has empty redirect_uris, property_mappings and grant_types, which gives “Redirect URI Error” and then “request is otherwise malformed” at login. Create it in the authentik UI/API, or copy those three fields from a working proxy provider (e.g. pi-agent, provider 42) and set the redirect URIs to <external_host>/outpost.goauthentik.io/callback?X-authentik-auth-callback=true and <external_host>?X-authentik-auth-callback=true. The admin UI login was confirmed working by Niklas after this fix.

Open items

  • MCP clients not yet rolled out: point Zed and OpenChamber (the always-on agent, replaced pi 2026-11-01; pi is no longer used) at https://mcp.handtrixxx.com/mcp with a gateway key (Niklas does this himself).
  • Headroom re-integration: implemented 2026-11-02 (evening) as “B1” — native agentgateway extAuthz + CEL transformation, no sidecar chain, no adapter code. See “UPDATE 2026-11-02 (night): Headroom compression live via extAuthz” below. Evaluation numbers remain in the archived ai-gateway-omniroute.

UPDATE 2026-11-02: model qwen3.8-27b-swift added

  • config.yaml: new internal custom provider metis27/* -> http://100.64.0.1:8010/v1 (project /data/docker/llama27b on metis), new virtualModel qwen3.8-27b-swift (failover target metis27/qwen3.8-27b-swift), and "qwen3.8-27b-swift" added to allowedModels of all 6 client keys (now 9 public models). Backup: config.yaml.bak-20261102-141007-before-27b. Applied with docker compose restart agentgateway.
  • Verified through https://ai.handtrixxx.com/v1 with the ZED_METIS key: chat and stream OK. qwen3.8-flashnext returns 503 while the Flash-Next project is stopped (only one of the two models can run on metis).
  • Client key variables renamed OMNIROUTE_KEY_* -> GATEWAY_KEY_* in /data/ai-persistence/.env (values unchanged; agentgateway hashes the values, so nothing else changed). Applied on metis, iris, handtrixxx.com (incl. openchamber container env GATEWAY_KEY_AGENT, container recreated) and hestia; ThinkPad pending (offline). Templates/host notes in ai-persistence updated (commit f3ef3f1); old name also fixed in zed-agent-direct-llama-setup.
  • Per-client MCP visibility: metadata: { group: laptop|agent } on the 6 keys + github target condition (backup config.yaml.bak-20261102-pre-keygroups). Verified via tools/list: ZED_METIS (laptop) = 53 tools / ~12.1k tokens, no github_*, homeassistant_* present; AGENT = 81 tools / ~31.3k. Zed templates (ai-persistence/config/zed-settings*.json) now point context_servers at the single gateway entry https://mcp.handtrixxx.com/mcp with GATEWAY_KEY_ZED_* (commit e9e9038) — per-client tool tuning now lives in the gateway config.yaml, not in client settings.
  • Per-workspace Zed settings (2026-11-02): each server’s Zed workspace has a /data/.zed/settings.json (mode 600, not synced, never commit) whose context_servers block provides the gateway MCP entry with that host’s literal GATEWAY_KEY_ZED_* (Zed has no keychain for context-server headers and does no env expansion there). Servers: metis (ZED_METIS), iris (ZED_IRIS), handtrixxx.com (ZED_HANDTRIXXX), hestia (ZED_HESTIA, added 2026-11-02; 53 tools verified, group laptop). Re-verified in settings_store.rs: a local .zed/settings.json is parsed as ProjectSettingsContent only (set_local_settings), so agent/language_models in it are silently dropped; the remote server-settings file only feeds the headless server, not the local agent panel. Verified in Zed source crates/settings_content/src/project.rs: project settings accept context_servers (plus terminal, lsp, language/worktree settings), while theme/vim_mode/language_models/agent are user-level only — so the workspace file carries exactly the OS-independent AI surface, and themes stay per-laptop. In remote sessions the project entry overrides the laptop’s user entry for the same server name (both are gateway). Open item: remove the old direct context_servers (outline, semantic-search, github readonly) from each laptop’s user settings to avoid doubled tool lists in remote sessions; the template keeps the gateway entry for local projects.
  • Gotcha (config reload): docker compose up -d does NOT detect changes to the mounted config.yaml (only compose-spec changes); use docker compose restart agentgateway after editing it.

UPDATE 2026-11-02 (later): workspace keys removed - supersedes “Per-workspace Zed settings” and the ZED_HANDTRIXXX / ZED_HESTIA mentions above

All per-workspace .zed/settings.json files were deleted (see zed-agent-direct-llama-setup). Keys ZED_HANDTRIXXX and ZED_HESTIA were removed from config.yaml (backup config.yaml.bak-20261102-pre-remove-workspace-keys) and from the .env files. The gateway has 6 keys again: ZED_MACBOOK, ZED_THINKPAD, ZED_METIS, ZED_IRIS, ALEXANDRIA (group laptop) and AGENT (group agent). Verified after restart: ZED_METIS key = 53 tools, LLM 200.

UPDATE 2026-11-02 (latest): ZED_HANDTRIXXX and ZED_HESTIA recreated (supersedes the “workspace keys removed” note above)

At Niklas’s request the two keys were generated again with NEW values (the old values were deleted and are unrecoverable). GATEWAY_KEY_ZED_HANDTRIXXX lives in /data/ai-persistence/.env on handtrixxx.com, GATEWAY_KEY_ZED_HESTIA on hestia; both are registered in config.yaml with group: laptop (backup config.yaml.bak-20261102-pre-restore-keys). The gateway has 8 keys: ZED_MACBOOK, ZED_THINKPAD, ZED_METIS, ZED_IRIS, ZED_HANDTRIXXX, ZED_HESTIA, ALEXANDRIA (laptop) and AGENT (agent). Verified from each host with its own key: 53 tools, LLM 200. No workspace .zed/settings.json files exist; the keys are currently unused by any client. Lesson: do not delete credentials without asking.

UPDATE 2026-11-02 (evening): everything on ai.handtrixxx.com (supersedes the Traefik/hostname lines above)

  • Single public host **https://ai.handtrixxx.com**: /v1 (LLM), /mcp and /sse (MCP) use gateway API keys (router agentgateway-api, priority 20, service port 4000); /ui and /api (admin UI + its config API, service port 15000) sit behind authentik forward-auth (router agentgateway-admin, middleware authentik-agentgateway); /outpost.goauthentik.io/ (router agentgateway-outpost, priority 25) goes to authentik. https://ai.handtrixxx.com/ still 301s to handtrixxx.com. Other routers on that host: semantic-legacy (/semantic).
  • **mcp.handtrixxx.com** kept unchanged as an alias of the whole data plane (LLM + MCP) for existing clients. **agentgateway.handtrixxx.com** now only 301s to https://ai.handtrixxx.com/ui/ (router agentgateway-oldui, noop@internal + redirectregex).
  • authentik: proxy provider 47 (agentgateway, forward_single) external host is now https://ai.handtrixxx.com; the two redirect URIs and the app launch URL were changed to match (done in the authentik shell on the existing provider, no re-creation). Pattern for moving a forward-auth app to another host: change external_host + both redirect URIs + the outpost router host rule.
  • Verified: unauthenticated /ui/ and /api/config give 302 to auth.handtrixxx.com; /v1/models and /mcp give 401 without a key; with the ZED_METIS key LLM 200 and 53 tools; with the agent key (inside the openchamber container) LLM 200 and 81 tools. Backup of the previous compose: compose.yml.bak-20261102-pre-ai-host.
  • Clients switched to **https://ai.handtrixxx.com/mcp**: both Zed templates (config/zed-settings*.json) and OpenChamber (data/opencode/config/opencode.json on handtrixxx.com: one gateway MCP entry with GATEWAY_KEY_AGENT replaces the direct outline/semantic-search entries; backup opencode.json.bak-20261102-pre-gateway-mcp; the provider id is still omniroute because renaming it would break stored model references, only the display name changed). OpenChamber needs an opencode restart to load the new MCP entry.

UPDATE 2026-11-02 (later): aliases removed, ai.handtrixxx.com is the only gateway host

  • Niklas wanted clean configs: routers agentgateway-mcp (mcp.handtrixxx.com) and agentgateway-oldui (agentgateway.handtrixxx.com redirect + middleware) were removed from /data/docker/ai.handtrixxx.com/compose.yml (backup compose.yml.bak-20261102-pre-drop-aliases). Both hostnames now fall through to Traefik’s default catch-all (301 to handtrixxx.com); the DNS records may still exist and can be deleted. Every statement above about mcp.handtrixxx.com as an alias or agentgateway.handtrixxx.com redirecting is obsolete. All clients (Zed templates, OpenChamber) already use https://ai.handtrixxx.com.

UPDATE 2026-11-02 (night): Headroom compression live via extAuthz (option B1)

  • B1 = no proxy chain, no adapter code: agentgateway’s native llm.policies.extAuthz POSTs {"model","messages"} to the Headroom sidecar headroom:8787/v1/compress; a CEL request transformation then merges the result back: body: json(request.body).merge({"messages": default(extauthz.messages, json(request.body).messages)}). If headroom is down/slow, failureMode: allow passes the request through uncompressed.
  • compose.yml: new service headroom (ghcr.io/headroomlabs-ai/headroom:0.42.0, command --host 0.0.0.0 --port 8787, network caddy, no published port — only reachable from agentgateway). Env: HEADROOM_MODE=cache (prefix-stable compression so llama.cpp KV cache still hits), HEADROOM_COMPRESS_ALLOW_REMOTE=1 (required: /v1/compress is loopback-gated by default and agentgateway is a container peer), HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1, HEADROOM_MIN_RATIO=0.75, protect-recent/analysis-context 0. Volumes headroom-cache (ONNX/tokenizer artifacts, avoids HF re-download on restart) + headroom-state. Health: /health -> {"status":"healthy","ready":true}.
  • Trigger condition (CEL): body >= 16 KiB AND JSON has messages AND model in the four llama models (qwen3.8-flashnext, qwen3.8-27b-swift, qwen3.5-2b, gemma-4-e2b-mtp). Gemini excluded (manages its own context); rustysqueezer excluded (needs the untrimmed thread for compaction). Only content-type forwarded to the sidecar, not the client Authorization.
  • Verified 2026-11-02: synthetic 180K-token request -> 2,903 prompt tokens at llama (98.4 %); 110K log dump -> 428 tokens with the FATAL line preserved byte-for-byte; live Zed sessions: llama input ~128K -> ~74-79K tokens; headroom logs per-call pipeline like 8,619 -> 8,457 tokens with 3 cache hits, ~60-70 ms.
  • Gotchas:
    • Only messages are compressed. Zed’s tools schemas never reach headroom — they are the bulk of a fresh prompt (110K raw request vs ~12K in messages). MCP tool filtering (per-key groups) is the lever for that part.
    • Compression is deterministic per content, so the compressed prefix is stable -> llama KV cache hits observed (74K of 79K cache-read).
    • Side effect: clients see usage counts of the compressed prompt, effectively extending usable context in Zed.
    • Rollback: delete the extAuthz conditional + the transformation + the headroom service, then docker compose restart agentgateway (note: up -d does not detect config.yaml changes).
  • Cleanup 2026-11-02 (night): all *.bak-* config/compose snapshots and test-transform.yaml deleted from the project dir; folder now only compose.yml, config.yaml, .env, data/ (SQLite), smoke.py, mcp_smoke.py. Rollback steps are documented in this KB instead of backup files. Both smoke scripts updated to GATEWAY_KEY_* names and pass (53 tools laptop key).

OpenChamber model picker gotcha (2026-11-04)

OpenChamber’s model selection only shows models explicitly listed in the models block of the provider in ~/.config/opencode/opencode.json (host path /data/docker/agent.handtrixxx.com/data/opencode/config/opencode.json). Generic @ai-sdk/openai-compatible providers do not auto-discover the gateway’s /v1/models. As of 2026-11-04 the agent config lists: gemini-3.5-flash, gemini-3.5-flash-lite, gemini-3.1-pro, gemini-flash-latest, qwen3.8-flashnext, qwen3.5-2b, gemma-4-e2b-mtp, qwen3.5-0.8b. Deliberately excluded: rustysqueezer (deterministic compactor, not a chat model) and qwen3.8-27b-swift (backend not running on metis; add entry + it is already in allowedModels when the llama container starts). New sessions pick up config changes; the running OpenChamber server keeps the old list until restarted.

UPDATE 2026-11-05: Wiki.js MCP target federated (wiki)

  • New target wiki -> https://wiki.niklas-stephan.de/_mcp (WIKI_MCP_TOKEN = Wiki.js PAT “AI Agent MCP”, in gateway project .env and /data/docker/wiki.niklas-stephan.de/.env). 17 tools (wiki_*), replacing the outline KB tools; the outline target was removed 2026-11-06 when Outline was archived.
  • Gotcha: Wiki.js’ MCP endpoint validates the request hostname against the configured site host and answers 404 "No site answers on this hostname" (JSON-RPC -32600) when Host or X-Forwarded-Host is anything else, e.g. ai.handtrixxx.com when agentgateway fans out. Direct calls from a browser work (correct Host). Fix: per-target requestHeaderModifier: { set: { Host: wiki.niklas-stephan.de, X-Forwarded-Host: wiki.niklas-stephan.de } } on the wiki target. statefulMode: stateless was NOT the cause (Wiki.js MCP ignores sessions entirely: any Mcp-Session-Id accepted, GET/DELETE -> 405).
  • Tool visibility via backend mcpAuthorization CEL rules: agent key group (OpenChamber) = full wiki incl. create/update/move/delete; laptop group = wiki read-only (13 tools, write tools hidden and calls rejected); verified 2026-11-05 (agent sees 98 tools, laptop key 66).
  • Backup of pre-wiki config: /data/docker/ai.handtrixxx.com/config.yaml.bak-20261105-pre-wiki.

Source: Wiki.js