niklas‑stephan.de

Pi 5 hestia as an aux LLM: measured speeds and model choice

Last updated 05/11/2026 · hestia, raspberry-pi, llm, llama-cpp

Hestia hardware (measured 2026-10-26)

Raspberry Pi 5, 8 GB (7.7 GiB visible), 4x Cortex-A76. Running restic-server, backrest-hub, arcane-agent, tailscale-hestia; ~1.0 GiB used, ~6.8 GiB available. LAN 192.168.68.54, tailnet 100.64.0.4. SSH from metis: ssh handtrixxx@192.168.68.54 (no key from handtrixxx.com yet - host key had to be accepted).

Measured on hestia (official llama.cpp b11910 prebuilt bin-ubuntu-arm64, Qwen3-4B-Instruct-2507 Q4_K_M, -t 4 -c 4096, 256-token gen)

  • prompt eval 16.6 tok/s, generation 3.2 tok/s.
  • Gotchas: Ubuntu Pi needs sudo apt install libgomp1 for the prebuilt binary; --no-display and -ot are not valid args in this build.
  • Model cached at /data/models/qwen3-4b-q4k.gguf (2.5 GB, SD card). /tmp/llama/llama-b11910/ has the binaries.

Why Pi 5 is memory-bound (docs, cross-checked)

Decode re-reads the whole weight set per token -> limited by LPDDR4X bandwidth, not cores. So more threads is slower for decode: llama.cpp docs/benchmarks consistently find -t 2 fastest, -t 3 ~2.84, -t 4 slowest; use -tb 4 for prefill (compute-bound). Overclocking 2.6 -> 2.8 GHz gives ~+6 % prefill but only ~+1 % generation.

Best models for Pi 5 8 GB (published benchmarks)

  • Qwen3.5-4B Q4_K_M + MTP head (github.com/marcofariasmx/raspberry-pi5-llm, llama.cpp, native -mcpu=cortex-a76): 5.6-5.8 tok/s short context, 2.58 tok/s at ~16k context, prefill 12-14 tok/s idle / 5.86 tok/s under load. MTP (--spec-type draft-mtp) +71-93 % but no public GGUF ships the MTP head - it must be converted (scripts/build-qwen-mtp.sh). Flags that worked: -c 32768 -np 1 -ctxcp 4 -cram 768 -t 2 -tb 4 -fa on -ctk q8_0 -ctv q8_0 -to 7200.
  • Qwen3-4B Q4_K_M: ~5-7 tok/s (actian.com), our 3.2 tok/s cold single-shot.
  • Qwen3-8B Q4_K_M: 2.15 tok/s, 5.2 GB RAM, -c 4096 is the safe ceiling (github.com/NikoEller/raspberry-pi-local-llm-lab). 8B fits but is not comfortable alongside the backup services.
  • Fastest / lightest: gemma3:1b, qwen2.5:1.5b, Qwen3-1.7B; BitNet b1.58-2B-4T ~8 tok/s with the smallest RAM footprint (needs bitnet.cpp). Avoid smollm2:1.7b (>5 GB).
  • Rejected: --no-repack frees ~2.5 GB but costs ~38 % prefill (real-task regression); GGML_LTO=ON no gain; build with -j2 not -j4.

Limits that decide the use case

  • Prefill is the killer for tool-heavy prompts: a ~7k-token tool schema costs ~10 minutes before the first token; a trivial 2-step agent task took 844 s; 1/5 on an AIME subset where a 27B scored 5/5. Do not use hestia as a coding agent - use it for prefill-bound work: embeddings, reranking, classification, summarisation, log triage, HA intents, drafts.
  • Bound **--cache-ram** (768 MiB on an 8 GB board). Left at the default 8192 MiB the server ballooned and hard-locked the Pi.
  • Decode halves as context fills (5.8 -> 2.58 tok/s): size any capacity estimate from the full-context number, not the empty one.
  • Thermal: sustained load drifts 6-8 %; throttling starts near 85 C; active cooling helps. vcgencmd needs /dev/vcio (absent on the Ubuntu image as of 2026-10-26).
  • Backup jobs and an LLM compete for the same 4 cores and the USB disk; cap threads to 2 if both run.

Integration if it happens

Same pattern as metis: compose project /data/docker/llama-aux/ on hestia, port e.g. 8001, reachable from metis over tailnet 100.64.0.4, optionally public via Traefik like llm.handtrixxx.com. Not yet built - no decision recorded.

Source: Wiki.js