niklas‑stephan.de

Always-on agent eval 2026-10: pi+pi-web-ui vs OpenCode vs LiteLLM Agent Gateway

Last updated 05/11/2026 · agents, litellm, pi, opencode, alexandria

Verdict (2026-10-25)

Evaluation finished. pi + pi-web-ui is the chosen always-on agent and is deployed and accepted at agent.handtrixxx.com; it fulfills all requirements (web chat/API, AGENTS.md + KB access, LiteLLM-routed models, scheduler/background tasks, sub-agents, PTY/SSH). The Alexandria MCP criterion is dropped: Alexandria is intentionally offline (see below), the agent reads the knowledge base via the read-only /kb mount instead.

Findings (verified 2026-10-25)

  • pi = earendil-works/pi (renamed from badlogic/pi-mono): minimal extensible agent harness, Node >= 22.19, unified multi-provider LLM API (OpenAI-compatible → LiteLLM works), RPC mode, reads AGENTS.md, no built-in sub-agents/permissions (containerize). UI: third-party pi-web-ui (xing-shuyin, MIT, v0.111) ships official docker compose (PI_WEB_TOKEN=... docker compose up -d, port 8787, mounts ~/.pi + ~/.pi-web), MCP hot-reload, scheduler/background tasks, sub-agents, PTY, remote SSH projects.
  • OpenCode: opencode serve (OpenAPI /doc, SSE, prompt_async) and opencode web UI; auth OPENCODE_SERVER_PASSWORD; native MCP + AGENTS.md. No scheduler; no official docker image. All prior instances (metis/iris/code.handtrixxx.com) removed 2026-10-15/16 — ask Niklas why before reinstating.
  • LiteLLM “agent” release = Agent Gateway (A2A): register A2A agents in admin UI or config.yaml agents: (needs >= v1.95; older key agent_list); governance (keys, budgets, logs, trace headers X-LiteLLM-Trace-Id/Agent-Id), agent_search virtual MCP tool, invoke via /a2a/{agent} or a2a/ model prefix; a2a-sdk >= 1.1. Also Python SDK litellm.agent(Harness.CLAUDE_CODE|CODEX|OPENCODE|DEEPAGENTS, task, sandbox=...). Use as governance layer in phase 3, not as the agent.
  • LibreChat (chat.handtrixxx.com) already had agents+MCP: quick zero-infra test bed, but no shell/filesystem/scheduler. (Retired 2026-10-25 - no longer running.)

Gotchas for wiring

  • llama on metis runs -np 1: give the agent a hosted LiteLLM model (gemini-class), not llama, to avoid blocking Zed sessions.
  • Alexandria Rust MCP enforces ALLOWED_HOSTS: add the agent container’s internal host if not using mcp.handtrixxx.com; add a dedicated agent-* bearer token to tokens.env.

Plan

Phase 0 decide (pi vs opencode, default model, public vs tailnet); phase 1 pi spike at /data/docker/agent.handtrixxx.com behind Traefik; phase 2 opencode fallback; phase 3 register agent via A2A in LiteLLM. Full plan: /home/handtrixxx/code/tasks/always-on-ai-agent-eval-2026-10.md

Deployed status (spike live 2026-10-25)

  • agent.handtrixxx.com running: project /data/docker/agent.handtrixxx.com/, container pi-web-ui (custom multi-stage Dockerfile; upstream compose does NOT work as-is: node-pty needs python3/make/g++ build stage on Linux; CMD must use explicit ./dist/... paths plus --import ./dist/server/resolve-global-sdk.js shim - pi SDK #260/#321). Traefik on caddy net, certresolver letsencrypt, priority 10.
  • Auth: PI_WEB_TOKEN (401 without, 200 via ?token= or cookie). The token value lives in /data/docker/agent.handtrixxx.com/.env (mode 600, not in /data/ai-persistence/.env); read it with grep '^PI_WEB_TOKEN=' /data/docker/agent.handtrixxx.com/.env | cut -d= -f2-. A 401 at the bare URL is expected, not a broken deploy. UI at /, headless via docker exec -w /workspace pi-web-ui /app/node_modules/.bin/pi -p --model litellm/<model> ....
  • Models: ~/.pi/agent/models.json provider litellm baseUrl https://ai.handtrixxx.com/v1, apiKey !printenv LITELLM_KEY_AGENT (do NOT use $VAR interpolation here - pi schema validation blanked it once; !command works). Virtual key agent-pi in LiteLLM (created via key/generate inside the litellm container), models qwen3.8-flashnext + gemini-3.5-flash(-lite). Spend is recorded in LiteLLM_SpendLogs (api_key hashed; qwen spend 0, local llama).
  • Context: /data/ai-persistence mounted read-only at /kb; /workspace/AGENTS.md present. Verified: qwen3.8-flashnext answered KB-only questions correctly with file citations (which host runs HA, kb location).
  • Gotcha hit: Gemini free-tier quota (5 req/min, gemini-3.6-flash) throttles the agent - prefer the qwen route or a paid Gemini key for agent sessions.
  • Scheduler = pi-web-ui UI feature (tools schedule_task/list/run/save/cancel, BgTasksModal); not exposed via REST, so scheduled runs are set up through the web UI. Accepted as the “always-on/proactive” requirement.
  • Alexandria MCP is offline ON PURPOSE (confirmed 2026-10-25) - it is not yet finally developed, so Niklas deliberately tore it down: no container, no Traefik route (mcp.handtrixxx.com falls through to the homepage 301), project dir /data/docker/mcp.handtrixxx.com/ removed on handtrixxx.com. Images alexandria:1/alexandria:dev still exist on the hub; the old build dir /data/maintenance/alexandria-rust/ and the metis clone /data/sources/alexandria/ were deleted 2026-10-25 (canonical repo only: GitHub handtrixx/alexandria, working clone on the ThinkPad at /home/handtrixxx/code/alexandria). The agent’s mcp.json (pi-home) is already configured for https://mcp.handtrixxx.com/mcp with Bearer ${MCP_MEMORY_TOKEN}; do not try to reconnect it while Alexandria is offline. KB access works fine via the mounted /kb.

OpenChamber global rules + workspace (2026-11-01)

  • Global rules are generated, not hand-edited: mem render with host openchamber (hosts/openchamber.md in kb-agent/ai-persistence, kind: host, frontmatter skip-sections: Tool-call hygiene; Context headroom because those sections are local-Qwen specific) writes ~/.config/opencode/AGENTS.md (container) = /data/docker/agent.handtrixxx.com/data/opencode/config/AGENTS.md (host). Skills: ~/.config/opencode/skills is a symlink to the repo skills/ (env MEM_SKILLS_LINK).
  • Env in compose: MEM_HOST=openchamber, AGENTS_WORKSPACE_FILE=/home/openchamber/.config/opencode/AGENTS.md, MEM_SKILLS_LINK=..., GIT_AUTHOR_NAME/EMAIL=openchamber-agent. Without AGENTS_WORKSPACE_FILE, mem would write /data/AGENTS.md inside the container (wrong).
  • Workspace: clone of kb-agent/ai-persistence at workspaces/ai-persistence (host project dir), also mounted at /data/ai-persistence in the container (so the CLIs and skills resolve like on the other servers). Only that clone is mounted, never the metis/iris /data/ai-persistence with .env.
  • Sync: hourly host cron on handtrixxx.com (user crontab): docker exec openchamber python3 /data/ai-persistence/bin/mem sync >> .../data/mem-sync.log. Verified: “Already up to date” + render 6573 bytes. Old workspace rules file kept as data/old-main-AGENTS.md.2026-11-01.
  • Identity: git/Gitea user openchamber-agent (write collaborator on kb-agent/ai-persistence and handtrixxx/niklas-stephan.de), SSH key openchamber-agent@handtrixxx.com pinned with from= on the homelab hosts; unattended jobs never get approval prompts and must not change production.
  • Change the rules in the repo (hosts/openchamber.md or shared agents.md), mem lint, push; the container picks it up at the next hourly sync.

Source: Wiki.js