niklas‑stephan.de

AI gateway: OmniRoute (ai.handtrixxx.com) - replaced LiteLLM 2026-10-31

Last updated 05/11/2026 · omniroute, litellm, headroom, gateway, handtrixxx, llm, archived

Central OpenAI-compatible gateway for all LLMs. Replaced LiteLLM on 2026-10-31 (decision by Niklas; the 2026-10-30 evaluation had advised keeping LiteLLM, see ai-gateway-omniroute). LiteLLM history/rollback: ai-gateway-omniroute (archived).

Layout (project /data/docker/ai.handtrixxx.com/)

  • Containers omniroute (diegosouzapw/omniroute:3.8.52, single-port mode 20128, caddy net, healthcheck on /api/monitoring/health) + headroom-omni (headroom-proxy:0.41.0, built from ./headroom, internal only).
  • https://ai.handtrixxx.com/v1 -> Traefik router priority 20, OmniRoute API keys only (REQUIRE_API_KEY=true - never set false: with false and a non-loopback bind inference is open, real incident #13310). https://ai.handtrixxx.com/ -> Authentik forward-auth (middleware authentik-ai, same outpost pattern as before) and then OmniRoute’s own login. OIDC SSO is supported and configured (2026-10-31, correcting the earlier evaluation which only found Entra docs): login page button -> authentik app omniroute (OAuth2 provider, client_id omniroute, secret OMNIROUTE_OIDC_CLIENT_SECRET in the project .env, redirect https://ai.handtrixxx.com/api/auth/oidc/callback). The password login (OMNI_ADMIN_PASSWORD in the project .env) is kept as fallback until Niklas confirms SSO in a browser; disable with python3 /data/maintenance/omniroute-oidc/setup.py --password-only --disable-password (re-enable with --enable-password). The script is idempotent and also (re)creates the authentik side. LiteLLM-era provider pk 40 is only the forward-auth anchor, keep it.
  • OIDC gotchas: OmniRoute honours the token’s email only when email_verified=true, but authentik’s stock email scope sends email_verified: false - so the allowlist oidcAllowedSubjects=["handtrixxx"] matches sub, and the authentik provider has sub_mode=user_username. Settings keys oidcEnabled/oidcIssuer/oidcClientId/oidcClientSecret/oidcAllowedSubjects/oidcScopes/oidcDisablePasswordLogin are set via PATCH /api/settings and need currentPassword (the dashboard password) plus an operator session cookie (not an API key). Issuer = https://auth.handtrixxx.com/application/o/omniroute/. No SAML. Browser login not yet verified end-to-end by an agent (server side verified: discovery, JWKS RS256, authorize 302, login redirect). Only /v1 is routed publicly; the old root aliases /chat, /models, /responses, /embeddings, /health are no longer exposed.
  • Tailnet-only admin port 100.64.0.2:8082. State in ./data/storage.sqlite (providers, keys, combos; encrypted with STORAGE_ENCRYPTION_KEY). JWT_SECRET, API_KEY_SECRET, STORAGE_ENCRYPTION_KEY must never change. Secrets only in project .env (mode 600). Back up ./data.

Models (names unchanged from LiteLLM)

qwen3.8-flashnext, qwen3.5-2b, gemma-4-e2b-mtp, rustysqueezer, gemini-3.5-flash, gemini-3.5-flash-lite, gemini-3.1-pro (= gemini-3.1-pro-preview), gemini-flash-latest. Each is a combo (single model, or for qwen: priority list) so /v1/models shows the bare names; the raw gemini/... ids also remain. Provider nodes (OpenAI-compatible, create via POST /api/provider-nodes {name,prefix,type:"openai-compatible",apiType:"chat",baseUrl,providerIdSuffix} then POST /api/providers; provider:"openai"+custom url does NOT work): metis -> http://100.64.0.1:8000/v1, headroom -> http://headroom-omni:8787/v1, auxqwen/auxgemma -> llama-aux containers, rsq -> http://rustysqueezer:8000/v1, plus the Google gemini provider (key stored in the DB). Embeddings are not served (none since 2026-10-27).

Headroom: verdict = KEEP, chained (decided 2026-10-31)

  • Deep evaluation 2026-10-31 (Niklas asked to test ALL compression options; scripts in **/data/maintenance/omniroute-compression-eval/**, README there). Verdict: keep Headroom; OmniRoute’s built-in compression stays OFF. Earlier claims “saves 0 tokens” and “header has no effect” were measurement artefacts (preview flattens messages; OmniRoute’s response cache returned identical replies for identical bodies; use a nonce). Method: live path (x-omniroute-compression: <named combo>, post-compression body read from /api/usage/call-logs/<uuid> requestBody), 24-message ~95k-char synthetic threads, 7 needle facts from OLD tool outputs (no real Zed bodies exist: LiteLLM spend logs stored none).
    • Lossless engines (session-dedup, headroom = SmartCrusher port, lite with lite.compressToolResults=false, ccr): 0 % on a Zed-shaped thread of plain-text tool outputs; on an agent-shaped thread with repeated file reads + a 400-row JSON array: session-dedup+headroom -52.6 % tokens, 7/7 facts. ccr needs the caller to expose omniroute_ccr_retrieve (Zed does not) and did nothing.
    • Lossy, rejected: lite with default compressToolResults=true (-82 %, lost facts; this was the auto-trigger default I had left on), aggressive (history aging rewrites old turns every turn: 3/7 facts lost, 0/7 prefix-stable), rtk (-66..78 %, but its command classifier mistook a cat x.yaml result for grep output and cut 2,804 chars to 427; 0/7 facts on the Zed thread, raising maxLinesPerResult/maxCharsPerResult did not help), ultra, caveman/standard (0 % on code/config text), llmlingua (-31 %, 2-3/8 facts, optional ONNX deps not installed), contextBudget dial (escalates into aggressive: 3/7 facts lost). read-lifecycle, ionizer, llm are not accepted as live pipeline steps in 3.8.52 (the API rejects them; only the preview endpoint takes them). omniglyph, codex-responses, Claude contextEditing not applicable (images / Responses API / Anthropic-only).
    • Real headroom-proxy 0.41.0, same threads, multi-turn replay (what matters with the llama.cpp prefix cache at ~500 tok/s prefill): --mode cache forwards 0 % on a single full-history snapshot (it freezes old turns) but over a growing thread cuts total sent 50 % and what llama must prefill 49 % on the agent thread (7/7 facts, prefix stable 7/7); ~4 % on the Zed thread. --mode token: 42 % but lost 1 fact, so cache mode stays. OmniRoute session-dedup+headroom: prefill 70 %, prefix-stable only 5/7 turns (an unstable turn = full re-prefill, ~60 s at 30k tokens).
    • Why Headroom wins: same lossless savings, plus deterministic prefix stability. Reconsider only if a combo is measured prefix-stable or the llama gets faster prefill. Fallback idea: a session-dedup+headroom combo on qwen3.8-flashnext (lossless).
    • State left (2026-10-31): master enabled:false, defaultMode:off, autoTriggerMode:off, only combo Standard Savings (session-dedup+lite). The per-request header only works while the master switch is on. Password login is now disabled (OIDC only), so admin-API scripts need a dashboard auth_token cookie from a browser session.
  • Topology: provider node headroom (chained proxy, --mode cache = llama.cpp prefix-cache friendly; --mode token saves more but breaks the prefix cache) -> metis llama (OPENAI_TARGET_API_URL=http://100.64.0.1:8000/v1). Combo **qwen3.8-flashnext** = priority [**headroom/qwen3.8-flashnext**, **metis/qwen3.8-flashnext**]: with headroom-omni stopped the request still succeeds via the direct route (tested; = LiteLLM’s fail_open). Gemini and aux models are not compressed (LiteLLM compressed every route). The per-request bypass header x-headroom-bypass no longer exists; call metis/qwen3.8-flashnext for an uncompressed request.
  • Caveat: headroom’s proxy pipeline in cache mode compressed 3 of 23 requests in the trial (synthetic threads); real Zed traffic share is unmeasured on this path. Check docker exec headroom-omni /stats (header Authorization: Bearer $HEADROOM_PROXY_TOKEN, container IP from docker inspect; host cannot resolve container names) after a few days and compare with the old 18.9 %.

Keys

  • Per client, restricted to the 8 model names above: OMNIROUTE_KEY_ZED_MACBOOK, _ZED_THINKPAD, _ZED_METIS, _ZED_IRIS, _ALEXANDRIA, _AGENT in /data/ai-persistence/.env (metis, iris, handtrixxx.com synced by hash 2026-10-31 on handtrixxx.com, metis, iris and hestia; ThinkPad not yet (offline); the pi agent also has OMNIROUTE_KEY_AGENT in /data/docker/agent.handtrixxx.com/.env, provider id gateway in its models.json). The old LITELLM_KEY_* values are dead (LiteLLM stopped); OmniRoute cannot import key values, so every client needed a new one.
  • Create: dashboard, or POST /api/keys {"name":...} (field is name, not label) with a session cookie, then PATCH /api/keys/<id> {modelAccessMode:"restricted",allowedModels:[...]}. The create response is the only time the value is shown.
  • Admin API: curl -c cj -d '{"password":"..."}' http://100.64.0.2:8082/api/auth/login, then use the cookie on /api/keys, /api/combos, /api/providers, /api/provider-nodes, /api/models/alias. Inference keys cannot manage (AUTH_001). Combos: POST /api/combos {name,strategy:"priority",models:[...]}.
  • Zed: laptops still need their keychain entry changed to the new key (URL unchanged) - see zed-agent-direct-llama-setup.

Behaviour verified 2026-10-31 (public URL)

No key / bad key -> 401; dashboard -> 302 authentik; every model answers; streaming with stream_options.include_usage, tool calls (id+function), RustySqueezer compaction text work; gemini-3.1-pro and gemini-flash-latest can hit Google quota/503 (same via direct Google; OmniRoute retries with a model cooldown, so a call can hang ~2 min - that is upstream, not the gateway). Response headers x-omniroute-* show provider/combo/latency.

Rollback to LiteLLM

/data/docker/ai.handtrixxx.com/litellm-retired/ has the old compose, litellm_config.yaml, env.litellm and a 74 MB litellm-db-20261031.sql; ./db is the untouched Postgres data dir. docker compose down, restore those files as compose.yml / litellm_config.yaml / .env, up -d; clients must switch back to LITELLM_KEY_*. The trial project ai-omni.handtrixxx.com and the Bifrost trial were deleted; the ai-omni DNS name is now unused.

Task-specific auto routes + free tiers (tested 2026-11-01, OmniRoute 3.8.52)

  • Built in: auto, auto/coding, auto/fast, auto/cheap, auto/smart, auto/offline and about 25 more variants (auto/chat, auto/reasoning, auto/coding:free, auto/best-free, …) are virtual combos scored over the CONNECTED providers (health, quota, cost, speed, task fit). Our 8 named combos are unchanged. Docs: cheaper-inference/OmniRoute (repo moved from diegosouzapw; site omniroute.online), docs/getting-started/AUTO-COMBO-GUIDE.md, docs/routing/STRICT_ZERO_COST.md.
  • Set 2026-11-01: setting freeAccessPolicy = strict (affects only auto/* pools: only models classified free with live quota headroom; paid Gemini key and metis are excluded from auto). Applied via PATCH /api/settings (no dashboard password needed with a session cookie). Key agent got auto, auto/coding, auto/fast, auto/chat, auto/reasoning, auto/smart added to allowedModels (other keys not changed yet). Revert: freeAccessPolicy = off.
  • Result: no usable keyless provider today. The strict pool contained only uncloseai (public beta closed, 401), so every auto/* call returned an error. Checked: Pollinations needs an API key now (401, free key at enter.pollinations.ai/keys); OpenCode Free (opencode-zen, can be added with any non-empty key) answers 429 FreeUsageLimitError upstream (only for the OpenCode client); OVHcloud AI Endpoints works only with NO Authorization header, but OmniRoute requires a key and sends it (403 with a dummy key); AI Horde = queue of minutes, no tool calls; DuckDuckGo/Cloudflare playground = ToS avoid / Playwright. The OpenCode connection I added for the test was deleted again.
  • Next step needs accounts/keys (free tiers, prompts are logged/used by the provider): Groq, Cerebras, NVIDIA NIM, Mistral, LLM7, Pollinations. Privacy rule: auto/* free routes only for non-sensitive tasks (no infra details, no secrets, no private file contents); enforced by rule text, not by a second key.
  • Admin access: password login is disabled (OIDC only); an operator cookie auth_token (log in via browser, copy from dev tools) works on the tailnet admin port http://100.64.0.2:8082/api/*. Niklas stored one at /data/maintenance/omniroute-session.cookie (mode 600); delete it when done.

Source: Wiki.js