Engine/model benchmark 2026-10-24: strix-llama -ub 512 wins, now prod

Last updated 05/11/2026 · llm, metis, benchmark, llama-cpp

Outcome of the 2026-10-24 engine/model A/B (guide: /data/docker/llama/EXPERIMENTAL_IMAGE_BENCH_GUIDE.md). Winner: strix-llama engine with **-ub 512**, now the standard **/data/docker/llama/compose.yml**. Rollback: variants/vulkan-radv.yml (the previous prod).

Final comparison (quiet box, same model/flags: GSQ-RCO Q2_0 + MTP + mmproj, -c 131072, KV q8_0, cache-ram 8192)

Why -ub 512: the extra ~4.7 GiB GTT of strix-llama was the compute buffer (about 5.4 GiB at -ub 1024; -ub 4096 = 21.6 GiB). Halving -ub gave the memory back and cost only ~16% prefill vs -ub 1024. Decode is faster up to 32k but ~4% / 9% slower at 64k / 100k. -ub 256 was never run (not needed).

Rejected / not viable (do not re-test without a new reason)

• REAP-288 (pruned 288 experts): prefill +55% but decode -41%, eval 6/9.
• TheRock nightly: fast prefill, decode collapses at >=64k. hrx-staging: no gain.
• Swift-1.5 GSQ-RCO: roughly tied with prod, no reason to switch.
• Gyro-S TQ1_0: TQ quant unsupported on vulkan-radv; on strix-llama it loads but crashes at prefill (GGML_ASSERT(nb10 % sizeof(src1_t) == 0) in binbcast.cu).
• Flint UD-Q6_K_XL (40B-A2.5B, text-only, no MTP): fast on vulkan (56 t/s decode, 1430 t/s prefill, 36 GiB GTT) but no vision, thinking control fails, eval 6/9; does not load on strix-llama. Fine as a text-only coding model if ever wanted.
• ignite-llama + DFlash2: hangs on both REAP and prod model (DFlash2 hyper-connection stream mean not implemented in llama.cpp, issue #30887). engramhalo: decode 35/27 t/s, worse.
• Ornith-122B: ~10B active, no MTP, so no speed gain; quality is not the problem. Download dropped.
• Not llama.cpp-compatible or too big for 56 GiB GTT: Gufo, PaoAI engine, halogen, ds4-server, SGLang+DFlash2.
• Lowering **--cache-ram** to save RAM was ruled out by Niklas (loses long-session cache hits, ~49% on OpenClaw).

Method + gotchas

• Suite /data/docker/llama/bench/: run_suite.sh, stages/NN_name.conf, summary in /data/sources/llm-bench/logs/imgbench/run_*/summary.md, live log status.log. Env: STAGES="name ...", RESTORE_PROD=0|1, SWAP_MAX_MIB, BASELINE_DIR. Watchdog aborts at GTT 52 GiB / swap limit. FULL=auto|always|never is set inside the stage conf and overrides the env var (stage 23 had FULL=never; edited to auto).
• Measure on a quiet box: background downloads inflate anon RAM and make paging/refault numbers misleading (first strix-llama runs showed 129 MiB free RAM partly because of this; the quiet re-run still showed 129 MiB at -ub 1024, so the compute buffer was real).
• Prod compose had -lm mmap --lazy-mode on; strix-llama uses -lm mmap --lazy-mode on-direct with -fit off -dev ROCm0.
• Compose project name: files in variants/ default to project variants; both files now set name: llama, so up -d of either file replaces the other. If a llama-flashnext container from another project exists, down it first (name conflict).
• Prod MTP head (Q8_0) only works with the matching Flash-Next base; other models need their own draft.
• Dead configs left in /data/docker/llama/variants/: ornith.yml, qwen27b.yml, test-8130.yml point at model paths that no longer exist on disk, so they cannot run; only vulkan-radv.yml is a usable rollback. Not deleted (tiny) - delete if a future cleanup touches configs.

Open items

• Verify in a long live Zed/OpenClaw session (58k-83k tokens) that decode and cache behave; if decode at depth disappoints, variants/vulkan-radv.yml is one command away.
• DONE 2026-10-25 (Niklas approved): losing model dirs deleted. Removed models/swift-1.5-q2_0 (63G), reap-288 (59G), gyro-s-tq1_0 (58G incl. mmproj + draft), flint (34G), dflash2 (341M). /data/docker/llama/models is now 71 GB: only qwen3.8-flashnext (prod base + MTP + mmproj) and mtp-hc (3.9G, unused by prod, kept for the engram-hc idea). Also docker rmi’d the bench engine tags (hrx-staging, rocm-10.0-engramhalo, therock-nightly, vulkan-radv-b11631-20261016, vulkan-radv-new, vulkan-radv-performance, local/ignite-llama:rocm10, ~13.9 GB) and docker builder prune (6.9 GB); prod rocm-10.0-strix-llama + rollback vulkan-radv kept. /data used 525 GB -> 297 GB; /v1/health ok after. Re-testing a rejected candidate means re-downloading its GGUF. The bench stage confs (stages/*.conf) still reference the deleted paths and will fail if re-run - that is intentional, they document the rejected candidates.
• Headroom compression was live under LiteLLM (archived ai-gateway-omniroute) and confirmed on the qwen route (23 events / 24 h, avg 33 % token reduction, cache reuse still good). Since 2026-10-31 the gateway is OmniRoute with Headroom chained — current state in ai-gateway-omniroute.
• strix-llama is 0.6.0-dev and auto-rebuilt upstream (unpinned tag): re-check llama-server --version, /health and a short bench after image pulls.

Source: Wiki.js