LLM Benchmark Suite on Metis (/data/sources/llm-bench) + Strix Halo BIOS memory

Last updated 05/11/2026 · migrated-from-outline, llm, metis

.

SUPERSEDED 2026-10-18/19: prod now runs llama-flashnext (Qwen3.8-Flash-Next Q2_0 + MTP, 43-50 t/s) — current state, flags and measurements in llm-serving-metis. The “final picture” below is the 2026-10-12 Ornith-era snapshot; this doc stays valuable for the benchmark methodology, scripts and the BIOS/TTM memory notes.

PICTURE 2026-10-12 (historical):

• Winner and prod config: Ornith-1.5-35B-A3B Q4_K_M, -c 131072 -np 1 -b 1024 -ub 1024, KV q4_0, embedded chat template, thinking cap 4096, MTP self-draft **--spec-type draft-mtp --spec-draft-n-max 2**. Prod llama is up on :8000. Details and rollback: “Local LLM Serving on Metis (llama.cpp)”.
• Why Ornith: eval 8/9 like the 27B, 2.4x faster wall-clock (283 s vs 673 s), 2.8x decode at 98k, 3x faster cold start. Full speed / quality / total rankings: /data/tests/llm-bench/RESULTS.md.
• Variant tests (run_variants.sh, **RESULTS.md** section “Ornith variant tests”):
  • Embedded template kept: the vendor template returns HTTP 500 on 3 of 13 edge cases (two system messages, system after user, no user message), same eval quality.
  • Thinking budget 512/1024/2048/4096: quality flat within noise (6.0-6.7 of 9), time per task 13.5 to 36.5 s. 4096 stays as a server-side safety cap.
  • MTP: plain ~76 t/s, n-max 3 helps at short context, n-max 2 is the best setting (90-94 t/s short, 102-104 on the prod coding prompt).
• n-max 2 context sweep on prod (cold decode t/s, plain in brackets): 8k 88.6 (73.8), 32k 83.6 (68.4), 64k 69.7 (61.5), 100k 59.5 (55.9), 120k 56.2 (51.9). Cold prefill 6-10 % slower, GTT 22.9 GiB (+1.6 GiB). Realistic expectation: 70-90 t/s in normal sessions, ~56 t/s at 120k. The old setup gave ~15 t/s in long sessions.
• Quality complaint check 2026-10-13 (replay A/B, **RESULTS.md** section “Quality complaint check”): the user found Ornith answers in Hermes Desktop mostly wrong. The failing prompts (real Hermes sessions) were replayed against Q4_K_M, Q6_K and the 27B+DFlash, n=5 per mode. Correct on the two context-reading cases A+B, of 20: Q4_K_M 11, Q6_K 8, 27B 15; the pure-knowledge case C is 0/5 for every model. Higher quant does not help (Q6_K slower, no better; Q8_0 skipped), the 27B is best but 5-14x slower and Ornith’s result flips with think/nothink. Decision: Ornith Q4_K_M + MTP n-max 2 stays. Harness problems (cwd override, date-only prompt) were fixed separately, see the Hermes doc. Tools: replay_ab.py, replay_model.sh (port 8124), replay_all.sh; a full 3-model window keeps prod down ~50 min.

STATE 2026-10-12 (end of benchmark): Benchmark finished, winner Ornith-1.5-35B-A3B Q4_K_M — production compose.yml switched to it (see “Local LLM Serving on Metis”). Rankings moved out of the suite README into /data/tests/llm-bench/RESULTS.md (speed / quality / total + verdicts). Xing/ROCmFPX removed entirely (scripts, fork build, host-runner glue in run.sh; lib.sh keeps an unused host branch). Weights deleted (~137 GB): Qwen3.6-35B, Saluki, Swift, ISTA 27B (ranked below 4th), Flash-Next Coder, Xing (untested). Kept: Ornith, Gemma + MTP drafter, Flash-Next Q2_0, 27B cache, dflash/, mtp/. Model scripts for deleted rows remain (./run.sh <model> --download). Sections below that mention the deleted models/Xing as present are historical.

Speed-benchmark suite for candidate primary LLMs: **/data/sources/llm-bench** (relocated from the old /data/tests/llm-bench; path verified 2026-10-24; the historical path mentions below are stale). README.md there is the full procedure. One script per model, logs in logs/, ./compare.py for the table. Reviewed + fixed 2026-10-12.

Two measurement paths

• ./run.sh <model> --bench — llama-bench, plain model, no speculative decoding. Tests: pp512, pp16384, tg128 (-p 512,16384 -n 128).
• ./run.sh <model> --serve-bench — llama-server with SERVE_ARGS + server_probe.py (reads the timings block: decode t/s, ~16K prefill t/s, draft acceptance). This is how the 27B + DFlash2 baseline is measured.
• run_all.sh = --bench for all four models + --serve-bench for the 27B (SERVER_BENCH=all|"" to change).

Gotchas (all verified against the real binaries)

• **llama-bench** rejects **-c** and every **--spec-\*** flag (invalid parameter for argument) — aborts the whole run. Keep those in SERVE_ARGS, not BENCH_EXTRA.
• -pg only reports the combined prefill+decode rate; use -p/-n lists for independent pp/tg numbers.
• -o jsonl rows have n_prompt, n_gen, avg_ts — no pp512 keys (the old compare.py regex matched nothing).
• Port 8123 is Home Assistant (host networking) — test server uses 8130 (LLM_SERVE_PORT).
• Symlinks into /data/docker/llama/models must be relative (ln -sfr): the dir is /models inside containers, an absolute host path dangles.
• Thinking-budget flag is --reasoning-budget (env LLAMA_ARG_THINK_BUDGET), not --thinking-budget.
• Don’t stack stop/restart of prod: run_all.sh sets LLM_BENCH_PROD_HELD=1; prod is only restarted if the suite stopped it. (Prod container llama may not exist/run — docker compose -f /data/docker/llama/compose.yml up -d.)
• HF downloads die with curl 92 (HTTP/2 CANCEL) — plain --retry doesn’t cover it; use --retry-all-errors.
• Xing (ROCmFP4) runs only on the host ROCmFPX fork (/data/tests/rocmfpx/build/bin, built 2026-10-11 @ cbc2cd5). Unverified: -fa 0 + -ctv q4_0 may abort at context creation (fallback BENCH_FA=1).

Strix Halo / Framework Desktop BIOS memory (verified 2026-10-12)

• Wanted: iGPU Memory Configuration = Custom, iGPU Memory Size = 512 MB, save + exit, plus kernel ttm.pages_limit=14680064 ttm.page_pool_size=14680064 (56 GiB GTT; kernel 7.0 uses ttm.*, not amdttm.*). Framework KB: “Changing Memory Allocation (AMD Ryzen AI Max 300 Series)”.
• Verify: sudo dmesg | grep -iE 'of (VRAM|GTT) memory' → 512M of VRAM, 57344M of GTT; bash /data/maintenance/checkram.sh → MemTotal ≈ 61-62 GiB. free -h does NOT show GTT.
• Wrong carve-out (48 GB seen): MemTotal ≈ 14.9 GiB; Vulkan lists a bogus 106496 MiB (VRAM+GTT summed); the HIP fork counts only free system RAM (~8.6 GiB) for UMA devices.
• llama-bench --list-devices (container + host fork) is a quick check: both should show ~56 GiB.

First smoke numbers (not the real run)

27B server probe with DFlash2: ~336 t/s prefill @16K tokens (≈ 48 s cold start), decode 23–41 t/s, draft acceptance 36–54% on code, ~100% on repetitive text.

Candidate review (2026-10-12)

• README now has “Expected results” (predictions before run), quality evidence, context ceiling, decision rule (quality gate via tools/eval.py -> time-to-solution -> headroom >=64k ctx).
• Missing strongest candidate: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF (IQ1_M dir only; expert-pruned 256/512, 6B active; shard1 ~29.6 GB resident + 28.8 GB n-gram mmap; vendor SWE-bench Verified 75.6, LCB v6 86.3). ~12 GB less resident than Q2_0. Not yet added (asked user).
• Optional extras: Flash-Next IQ3_XXS (47 GB resident, may fit now with 56 GiB GTT), Nemotron-3.5-Lightning-30B-A3B (ggml-org Q4_0, agentic quality unknown).
• Xing risks: needs ROCmFPX fork; KV ~0.4 MiB/token (x2 with MTP), KV quant ignored on arch -> ctx ceiling ~64-96k without MTP, ~32k with MTP; verbose/Chinese thinking. The “PPL 318,130 at -ub 512” note is about the imatrix build only (unquantized MUL_MAT_ID bug), not runtime.
• Skip: GLM-5.3-Flash, Gemma-4-26B-A4B, gpt-oss-120b, Devstral-Small-2, Mistral-Small-4-119B, ISTA 27B GSQ (user eval rejected), unvetted third-party fine-tunes.

Matrix change (2026-10-12, later)

• Added models/qwen3.8-flash-next-coder-iq1m.sh (ISTA Coder, dir IQ1_M/, shard1 29.6 GB + shard2 28.8 GB, no mmproj; -c 65536). Dir name misleading: retained weights ~3.5 bpw, so decode speed is NOT expected to beat Q2_0 — benefit is ~8 GB less resident memory.
• run_all.sh now takes MODELS="..." (default: 27B, 35B-A3B, Flash-Next Q2_0, Coder). Xing is opt-in (MODELS="xing4.0-29b-a4b-rocmfp4" ./run_all.sh): KV-heavy ctx ceiling, verbose thinking, fork-only. Files/fork build kept (download complete 16.6 GB).
• Suite still NOT run (user instruction).

Depth pass + revised decision rule (2026-10-12)

• User reports ~15 t/s in long conversations on prod (27B + DFlash2; ~30 t/s short) and is willing to trade some quality for a big speed gain. So tg128 (empty ctx) is not the yardstick.
• lib.sh:run_bench now runs a second llama-bench pass -p 0 -n 128 -d 16384,32768 (1 rep; BENCH_DEPTHS, BENCH_DEPTH_REPS) -> rows tg128@d16384 / @d32768 (compare.py already labels depth). llama-bench -d/--n-depth exists in the prod image (build 11541). Smoke-tested on 27B (tiny depths): fine. Suite ~40 min.
• Quality gate relaxed to a floor (~6-7/9, no token inflation); 35B-A3B is a live candidate again -> keep it in the default matrix. “Big gain” ~ >=2x the 27B’s long-session rate.
• run_all.sh writes logs/run_all.status (state, step N of M, current, finished steps + durations).
• Use tools/bench-depth.py --url http://127.0.0.1:8130 + tools/eval.py <label> --url ... on finalists (README step 5).

RESULTS of first full run (2026-10-12 08:57-09:18, 21 min, all 5 steps ok)

llama-bench, no spec decoding (t/s):

• Depth barely slows llama-bench decode (-6..-9% at 32k) -> user’s prod ~15 t/s in long sessions is NOT attention cost; suspect DFlash2 acceptance at depth / thinking. Still to measure: 27B+DFlash at depth (bench-depth.py).
• 35B-A3B ~2.1-2.4x the 27B+DFlash probe; Flash-Next Q2_0 ~= 27B+DFlash probe w/o spec decoding; Coder 12% slower than Q2_0 (no speed win, only ~8 GB less resident).
• Next: eval.py + bench-depth.py on 35B-A3B (speed winner) and 27B (reference). Production llama container still down after the run.

Quality + long-context pass (2026-10-12, ./run_quality.sh, ~76 min)

Setup: prod sampling (temp 1.0/top-p 0.95/top-k 20), thinking cap 4096, -c 131072, -np 1, KV q4_0, DFlash2 for 27B. Port 8130 (HA uses 8123). All three loaded at 128k; Flash-Next ~45 GiB RAM used.

Takeaways:

• 27B+DFlash2: best quality + token efficiency, weak at long context (decode ~19-20 t/s past 64k, cold 98k ~8 min) - matches observed ~15 t/s in prod (256k + thinking).
• 35B-A3B: ~2.4x faster at depth, 3x faster cold start, but ~2x tokens on reasoning (hits 4096 thinking cap) -> only ~13% wall-clock win on eval, 6/9 vs 8/9. Tool-heavy turns (little thinking) likely show full 2x+ but NOT measured.
• Flash-Next Q2_0: side-grade, not worth switching (839 s vs 673 s, 45 GiB RAM).
• Caveats: 9 tasks x 1 sample (noisy); math_coprime_pairs fails for all models; retrieval probe too easy; 256k not measured.
• Gotchas: llama-bench rejects -c/--spec-* (spec decoding only measurable via llama-server); Coder “IQ1_M” folder is actually ~3.5 bpw.
• Production llama container was left DOWN after the suite; restart with docker compose -f /data/docker/llama/compose.yml up -d (only on user’s OK).
• Possible follow-ups: 35B with lower thinking budget; tool-heavy turn bench; 256k test; MTP for Flash-Next; run 27B + 35B side by side (unverified fit).

Candidate re-check (2026-10-12): web + HF research for Strix Halo 64 GB

Engine: local kyuz0/amd-strix-halo-toolboxes:vulkan-radv = llama.cpp build 11541 (commit f2918cabb) — supports --spec-type draft-mtp (needs ≥ b11259 for multi-turn MTP, PR #29638). No newer image needed.

Shortlist (file sizes verified via HF tree API)

Dropped (with reason)

Ternary-Bonsai-2-27B (needs PrismML fork; PQ2_0/PTQ1_0 slow on Vulkan, garbled on HIP gfx1151) · bytkim/Qwen3.8-27B-pi (Pi-harness tuned, no numbers) · empero-ai Qwen3.8-35B-A3B-Distill (Qwen3.6 SFT, no agentic evidence; no official Qwen3.8 35B-A3B exists) · Ornith-1.5-9B (weak on long tasks) · GLM-5.3-Flash (320B, no stock support) · Kolibri-1 (needs patch) · Qwen3-Coder-Next (41 GiB, 40 t/s, no advantage) · gpt-oss-120b (59 GiB, doesn’t fit) · Nemotron-3-Nano (weak) · Mellum2.1 (sub-agent at best) · Flash-Next Coder (already tested, dropped).

Engine / backend findings

• Gufo runtime (ghcr.io/gufo-org/toolboxes/gufo-runtime): ~2x llama.cpp prefill on 27B (pp 387 vs 194 @65k) but same long-ctx decode; daily releases, open prompt-cache re-prefill bugs, one amdgpu crash report, no independent 64 GB data. Not recommended now. Vendor bench (128 GB box): 27B Q4 AR decode 12 t/s @d0, 8.5 @131k; DFlash2 26 -> 14.7 t/s — matches our observed ~15 t/s in production.
• Vulkan tuning (strixhaloguide): -b 2048 -ub 512 (ub > b is silently clamped), Mesa kisak 26.1.7 vs 25.2.8 = +17% pp512, --no-mmap, -fa 1.
• HIP: llama.cpp commit c7d87229 corrupts output on gfx1151 (issue #25992, closed by stale bot, not fixed). Avoid unpatched HIP; kyuz0 rocm-10.0 carries a patch.
• Vulkan(RADV) wins decode, ROCm wins prefill on MoE.
• MTP on dense 27B ~1.84x at n=3; only ~1.2x on 35B-A3B MoE (slb350).
• Caveat: eval.py has 9 tasks x 1 sample; 2-task gap = hint, not proof.

Round 2 results (2026-10-12, 128k ctx, thinking cap 4096, KV q4_0)

* -ub 512 (see below). Retrieval 10/10 everywhere. Ornith/Gemma/27B/Flash-Next fail only math_coprime_pairs (everyone does).

Verdict: Ornith-1.5-35B-A3B Q4_K_M replaces the 27B+DFlash2 as primary: same eval result and token economy, 2.4x faster wall-clock, 2.8x decode @98k, 3x cold start, cached agent turn 6-9 s vs 17-25 s. Recommended flags: -ctk q4_0 -ctv q4_0 -b 1024 -ub 1024 -c 131072 -np 1, --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0, LLAMA_ARG_THINK_BUDGET=4096, no spec decoding. Keep the 27B as fallback for hard math/reasoning.

Gotchas:

• -ub 2048 crashes the IQ2/IQ3 27B quants at ~64k prefill with vk::DeviceLostError (also with MTP off and checkpoints off); -ub 512 is stable. Debug by running the server by hand and docker logs (the suite deletes the container).
• Per-model sampling needs the QUALITY_POST hook in run.sh (QUALITY_COMMON comes last on the command line and would override per-model settings).
• Gemma MTP uses a separate drafter file (--spec-draft-model, mtp-gemma-4-26B-A4B-it.gguf); the Qwen3.8 27B GSQ/Saluki GGUFs embed the MTP head (--spec-type draft-mtp only).
• MTP on the 27B-class quants (n=3) was not better than DFlash2 at depth (20-25 vs 20 t/s @98k).

Not verified: Ornith’s embedded chat template vs the vendor’s modified one; tool-heavy multi-turn agent runs; 256k context; Ornith + MTP; Gemma’s single tool-call miss. 9-task eval = coarse filter.

2026-10-24 engine/model A/B suite (current)

The modular suite for engine/model comparisons lives in /data/docker/llama/bench/ (stages, run_suite.sh, logs /data/sources/llm-bench/logs/imgbench/); results and method in llm-engine-benchmark-2026-10-24. Outcome: strix-llama -ub 512 is prod.

Source: Wiki.js