niklas‑stephan.de

llama.cpp image source for metis and what to watch upstream

Last updated 06/11/2026 · metis, llama-cpp, llm

UPDATE 2026-10-24: prod now runs **kyuz0/amd-strix-halo-toolboxes:rocm-10.0-strix-llama** (build 12169) via the standard **/data/docker/llama/compose.yml**; the vulkan-radv text below is the rollback (**variants/vulkan-radv.yml**). Details: **llm-engine-benchmark-2026-10-24**. See **llm-serving-metis**.

UPDATE CHECK 2026-11-06 (desk research; nothing pulled, prod unchanged).

  • Prod: strix-llama build 12297, commit ee30b949e (fork master 2026-11-01); Hub tag rocm-10.0-strix-llama last rebuilt 2026-11-04 12:47 and prod already matches that rebuild - pulling today changes nothing. Check inside the container: /opt/strix/versions.txt (build_date, rocm_systems, llama_cpp).
  • Fork head is ea5285ddfc (PR #251, merged 2026-11-04): convert-only fix writing QSA compress ratios for qwen4exp indexed_attention layers; prod GGUF already has correct qwen4exp.attention.compress_ratios ([0,0,0,4,…]) - no effect for us.
  • Open fork PRs to watch for the next rebuild: #255 (faster MTP-draft prefill), #254 (MTP carry reset at pos 0), #253 (--spec-draft-mtp-vocab for qwen4exp), #252 (TQ2_T fused MUL_MAT_ID tile), #257 (ROCm WMMA fix for LLVM 23 / ROCm 7.16+).
  • Correction: prod binary supports trellis/TQ quants (see Gyro page) - the old “no TQ symbols” gotcha applied to the 12263-era image.
  • Model side: only Gyro-S (TQ1_0, 27.64 GiB weights) and Gyro-M (TQ2_0, 34.98 GiB) fit the ~56 GiB GTT ceiling (Gyro-L TQ3_0 50.21 GiB marginal); ThinkingCap (IQ4_XS 95.5 GB), ISTA GSQ-RCO IQ2_XS/IQ3_XXS (68-76 GB) and Swift-1.5 Q4_K_XL (~103 GiB) all exceed it. Gufo v0.31.0 (new rocm-10.0-gufo tag, rebuilt 2026-11-05) claims ~1,800 tok/s prefill / ~117 t/s MTP decode but officially only Unsloth UD-Q4_K_XL (~104 GiB) - does not fit metis; real Strix Halo reports (llmkube) say Gufo is ~7% slower single-stream decode than llama.cpp but serves multiple slots concurrently.

UPDATE CHECK 2026-10-30 (desk research only, nothing pulled or changed; Niklas declined the image update for now).

  • Prod: strix-llama build 12254 (commit a5cd12534, local image 2026-10-28). Hub tag rocm-10.0-strix-llama was rebuilt 2026-10-30 15:29 (newer than ours); mainline llama.cpp at b12015. Our engine tracks the fork halo-box/strix-llama.cpp, not mainline.
  • Already in prod: fork PR #213 (2026-10-27, GDN Z16 chunked prefill, MoE mmb, HC W8A8, QSA prefill; source of the big prefill gain). No new kernel-speed PR since.
  • Not in prod yet: fork PR #235 (merged 2026-10-29, rebase of #163) keeps the incremental QSA prefix across mrope images. Before it, one image in the cache forces masked QSA prefill (slow, buffer grows with ub x depth; same mechanism as llama.cpp #30032, crash at depth with large -ub). Also fixes the MTP draft after an mrope image. Relevant for vision + long sessions; a docker pull + recreate picks it up (then check /health, vision test, short bench).
  • Mainline issues: #30544 closed 2026-10-24 (Vulkan only); #30032, #30048, #30487, #30887 still open.
  • Untested model candidates: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF (IQ1_M, 27.6 GiB backbone + 26.8 GiB shard + mmproj; expert-pruned, code/agentic-tuned, SWE-bench Verified 79.2 vs 82.8 BF16, updated 2026-10-19) would free ~7 GiB GTT vs Q2_0 (35 GiB), room for -ub 1024 or bigger -c; risk: pruned REAP-288 lost 41% decode and scored 6/9, MTP head compatibility unchecked. IQ2_XS of the unpruned GSQ-RCO (36.5 GiB first shard, ~1.5 GiB over Q2_0) is a quality-for-memory option, benchmarked upstream only. Any test needs prod down (bench suite) and Niklas’ OK.
  • Gotcha: the vulkan-radv rollback image is NOT on disk (removed in the 2026-10-25 cleanup; docker images kyuz0/amd-strix-halo-toolboxes lists only rocm-10.0-strix-llama). variants/vulkan-radv.yml needs a docker pull first and will give a newer build than the old prod (unpinned tag).

Verified on metis 2026-10-20: /data/docker/llama-flashnext/compose.yml line 31 uses docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv (no digest/date pin). Running build: docker exec llama-flashnext llama-server --version -> b11723, commit 31288c9c6 (image created 2026-10-19). Upstream ggml-org/llama.cpp was at b11756 on 2026-10-20.

How the image is maintained: kyuz0/amd-strix-halo-toolboxes rebuilds the stable tags (vulkan-radv, rocm-10.0) automatically whenever llama.cpp master moves. Consequence: a docker compose up -d --force-recreate after a pull silently changes the llama.cpp build. Pin for reproducibility (metis already has a dated local tag pattern, e.g. vulkan-radv-b11631-20261016), keep the previous tag for rollback.

Check state cheaply:

docker exec llama-flashnext llama-server --version
docker images kyuz0/amd-strix-halo-toolboxes --format "{{.Tag}} {{.CreatedAt}}"
curl -sS https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=3

Upstream llama.cpp issues that apply to the live flashnext config (qwen4exp = Qwen3.8-Flash-Next, -fa on, KV q8_0, -c 131072 since 2026-10-21 (196608 before), vision + MTP), checked 2026-10-20:

  • #30032 (open) - qwen4exp QSA indexer materialises O(n_kv x ubatch) f32 buffers once any image is live in the KV cache; crashes at depth with large -ub/-b. The maintainer’s own mitigation is lowering -ub (8192 -> 2048 at a ~43 % prefill cost). Metis’ -b 1024 -ub 1024 is exactly the workaround - do not raise -ub (this is the real mechanism behind the “Vulkan hang/-ub 2048” caution in llm-serving-metis).
  • #30307 (closed/fixed) - qwen4exp + --spec-type draft-mtp aborted on image input after #30257 (b11548). b11723 is past the fix; rolling the image back breaks vision + MTP.
  • #30048 / #30487 (open) - Vulkan FA + quantized KV dequantises K+V to f16 into one scratch buffer whose combined size is not checked against maxBufferSize; over-allocation shows up as device loss. Relevant at 192k with q8_0 KV on a 56 GiB GTT box.
  • #30544 (open) - Vulkan skips F32-precision MUL_MAT_ID (mat-vec MoE down-projection), a MoE decode regression route; watch decode t/s after an image bump.
  • #30887 (open, 2026-10-20) - Qwen3.8-Flash-Next + DFlash2 crashes in extract_layer_inputs() on the Unsloth build; DFlash sidecars for Flash-Next are not trustworthy yet (MTP is the safe path).

Decision 2026-10-24 (Niklas): stay on the unpinned latest **vulkan-radv** - do not pin dated tags; upstream llama.cpp master moves fast and a recreate silently picks the newest build.

Other toolboxes in the same image repo (experimental, manual/poll builds; all tags verified live 2026-10-24) - potential experiments, not adopted: rocm-10.0-strix-llama (halo-box/strix-llama.cpp, retained-PM4 ROCr+HIP; published Flash-Next Q4_K_XL 1207 t/s pp2048, 1055 t/s at 32k depth, 43.9 t/s decode - much higher prefill than metis’ ~450-486 t/s, different harness and quant), rocm-10.0-gufo (Gufo engine, Flash-Next UD-Q4_K_XL + shared-Q8 MTP), vulkan-radv-performance (Nathanw1014/llama.cpp strix-halo-vulkan fork: FA/KV/MoE work), rocm-10.0-engramhalo (EngramHalo.cpp - keeps the Flash-Next n-gram table in CPU/SSD instead of GTT, i.e. aimed at exactly metis’ lazily-read 28.8 GB shard / page-cache problem; requires -lm mmap --lazy-mode on, 1 slot, -c <= 163840 with the MTP sidecar). New since 2026-10-21: vulkan-rocmfpx + rocm-10.0-rocmfpx (ROCmFPX/ROCmFPX HIP/Vulkan builds with custom ROCmFP I4/W4A4 and FP3/4/6/8 weight formats, MTP spec decode, agent-aware presets - needs FPX-quantized models, not compatible with our Q2_0 shards), therock-nightly (AMD TheRock gfx1151 nightly ROCm tarball) and hrx-staging (AMD’s HRX-enabled llama.cpp staging tree, ROCm/ggml-staging-automation). The vulkan-radv-performance fork (Nathanw1014/llama.cpp strix-halo-vulkan) last committed 2026-07-09 - the tag is rebuilt but the fork is stale. Deprecated: the old -mtp tags (MTP is now in mainline llama.cpp).

Automated A/B suite (built 2026-10-24, started by the agent with Niklas’ OK; Niklas: leave prod DOWN afterwards, he decides on switching): /data/docker/llama/bench/ (README.md, prefetch.sh, run_suite.sh, stages/NN_*.conf, summarize.py). 11 stages in order: baseline (prod re-measured), REAP-288 (+ngram-mod), engramhalo, strix-llama (+ub4096), prod+MTP+ngram-mod, Swift-1.5 Q2_0, Gyro-S TQ1_0, therock-nightly, hrx-staging; skipped: ThinkingCap/PaoAI/halogen (do not fit 56 GiB GTT), rocmfpx, radv-performance (stale). Gate (thinking, tools, 8k/32k retrieval+decode+prefill, vision) for every stage, full probe (64k/100k + eval.py) only if the gate beats the baseline. Tail: tail -f /data/sources/llm-bench/logs/imgbench/status.log; table: /data/sources/llm-bench/logs/imgbench/summary_latest.md (+ latest/results.csv). Watchdog aborts a stage at GTT > 52 GiB or swap > 2 GiB. Prod is left down at the end: restore with cd /data/docker/llama && docker compose up -d. Gotchas found: llama-bench via docker exec cannot work beside the running server (GTT), so speeds come from server timings; prod build b11853 has no --spec-ngram-mod-n-start (use --spec-ngram-mod-n-match); REAP-288 has no MTP head and no mmproj (our mmproj is reused); hrx-staging is build 10229 (no --reasoning-effort/--lazy-mode); engramhalo build 10807, strix-llama 12169, therock-nightly 11854. Results: not yet recorded here - add the table after the run.

A/B benchmark guide for the experimental tags: /data/docker/llama/EXPERIMENTAL_IMAGE_BENCH_GUIDE.md (written 2026-10-24). It exists because the Zed agent runs on prod llama and two instances cannot coexist (GTT 56 GiB) - the external runner tests with prod stopped, prod :8000 baseline numbers are in the guide, restore-prod step is mandatory. Prod build as of 2026-10-24: b11853.

Model-side candidates (HF Qwen3.8-Flash-Next tree, surveyed 2026-10-24, in the guide): REAP-288 expert-pruned GGUF (45+14 GB, ~44 GB VRAM, 3B active - best fit for unified-RAM pressure), Gyro-S TQ1_0 (54.5 GB, “98% of Q4 quality at half memory”, check TQ quant support in our build), ThinkingCap finetune (reasoning tokens -38.5%, IQ4_XS ~77 GB), Swift-1.5 GSQ-RCO (efficient-thinking, same GSQ family); skip NVFP4/MLX/EXL3/DFlash2; PaoAI STRIX and halogen-flash are 128 GB-class Strix Halo engines, watch only.

Candidate audit 2026-10-24 (agent, after Niklas asked to re-check exclusions; see standing rule in **llm-serving-metis**: engine is NOT limited to llama.cpp). Fit rule (was applied too bluntly before): the 26.8 GiB engram/PLE table stays on SSD in lazy/mmap mode, so resident GTT is roughly file size - 26.8 GiB + ~8 GiB (KV q8_0 at 131k, mmproj, MTP, compute buffers). Prod check: ~62 GiB main file -> 43.5 GiB GTT idle. Watchdog limit 52 GiB.

  • Wrong exclusions, corrected: vulkan-radv-performance is NOT stale (Nathanw1014 strix-halo-vulkan commits through 2026-10-23, merges a gyro branch, image rebuilt 2026-10-23; pure Vulkan drop-in -> stage 14). EngramHalo was not a dead end: our mtp-*-Q8_0.gguf is the deprecated pre-llama.cpp-PR-#28901 layout (no hyper-connection tensors; error output_hc_norm.weight not found); the fork needs mtp-Qwen3.8-Flash-Next-hc-Q8_0.gguf from EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF (4.1 GB, same weights; stored in /data/docker/llama/models/mtp-hc/) -> stage 13. qwen4exp “unknown architecture” on hrx-staging = stale image (built 2026-09-14, before qwen4exp support); AMD’s ROCm/ggml-staging-automation was bumped 2026-10-23, needs a local rebuild (Dockerfile.hrx-staging, build arg STAGING_BRANCH). FPX models for Flash-Next DO exist (ArtomYuan/Qwen3.8-Flash-Next-ROCmFPX 88-170 GiB; rcmorano/sh0wie-Qwen3.8-Flash-Next-REAP-288-ROCMFPX STRIX_LEAN 72 GiB / COHERENT 89 GiB).
  • Image freshness (Docker Hub vs local, 2026-10-24): strix-llama (10-23), therock-nightly, vulkan-radv (10-24) are current. rocm-10.0-engramhalo (Hub 09-03) and hrx-staging (Hub 09-14) are stale on the Hub; the EngramHalo branch gained mmap: drop page cache behind uploaded tensors (LLAMA_MMAP_DROP_BEHIND) on 2026-10-01 -> a local rebuild of the newer branch is worthwhile. rocm-10.0-gufo (Hub 10-24, Gufo engine: gufo serve ... llm --model ... --speculative mtp --mtp-model <shared-Q8>, port 18080) was never put on the candidate list - gap.
  • Borderline, not excluded (est. GTT): PaoAI STRIX Q2-1.0 GGUF (72.9 GiB, backbone 46.1 GiB, ~54.6 GiB est. in our images; claims 1375-1717 t/s prefill, 40-63 t/s decode on a 128 GB box); REAP-288 FPX STRIX_LEAN (72 GiB, ~53 GiB, needs rocmfpx image). Both are last-resort memory-limit experiments only. Its own engine ghcr.io/guevae2/paoai-engine:1.0.0 (:8080, PAOAI_CTX) reports 70.7-72.2 GiB GTT -> does not fit metis.
  • Confirmed exclusions: ThinkingCap GGUFs are 89 GiB (IQ4_XS), 105 (Q4_K_M), 160+ (earlier note said 77 GB - wrong) -> ~62+ GiB resident; Baekpica ThinkingCap MQ-Q5 (77.5 GiB) targets ds4-server (CUDA) with the whole backbone GPU-resident; halogen-flash v2 62 GiB / ht43 54 GiB resident (own .hgn format + engine ghcr.io/peonist-ai/halogen-flash-server) > 56 GiB GTT; DFlash2 GGUFs: model card says llama.cpp cannot drive the draft yet (hyper-connection mean not computed; use SGLang), issue #30887 still open; NVFP4/FP4/MLX/EXL3 not usable on gfx1151/llama.cpp.
  • Suite findings so far (run_20261024_095221, prod stopped): baseline decode 53.5/49.8/48.6/43.7 t/s at 8k/32k/64k/100k, GTT peak 44.2, eval 8/9, MTP acceptance 91-96%. TheRock nightly: prefill +17-33% (332 vs 271 t/s at 32k) but decode collapses deep (36.1 @64k, 29.6 @100k) and eval 6/9 -> lukewarm. strix-llama (--load-mode none): screen prefill16k 524 vs 382 t/s (+37%) but watchdog abort (GTT 48.7, swap 2.6 GiB) -> retry with -lm mmap (stage 12). strix -ub 4096 needs a 21.6 GiB compute buffer, does not fit. base+ngram-mod: decode@32k 38 vs 50, reject. EngramHalo failed only for the sidecar reason above. HRX: stale image, see above. Harness bugs fixed on the way: summarize.py screen was missing, Pillow was not installed (python3-pil now), EXTERNAL_CN unset. Baseline paging numbers are noisy (downloads running in parallel); majflt ~10k/s even on prod.
  • Evaluation under the engine-agnostic rule (same day): widening the engine set does not open new options, because the binding constraint is the 56 GiB GTT / 62 GB RAM, not llama.cpp. Backbone sizes (file minus the 26.8 GiB engram shard): prod GSQ-RCO Q2_0 35.0 GiB (-> 43.5 GiB GTT idle, i.e. ~8.5 GiB for MTP 3.9 + mmproj 0.8 + KV/compute), PaoAI Q2-1.0 46.1 GiB (-> ~54.6 GiB est. on prod-style flags, too tight vs the 52 GiB watchdog), Unsloth UD-IQ1_S/IQ1_M 40.8/42.6 GiB (IQ1 quality), UD-Q2_K_XL 46.7 GiB (~55 est.), IQ3_XXS 49.5, Q4_K_XL 76.9. Non-llama.cpp engines checked: Gufo supports only Unsloth UD-Q4_K_XL (103.7 GiB, 4 shards) + shared-Q8 MTP, memory model built for 128 GiB -> does not fit and cannot load our Q2_0; PaoAI engine vendor table reports 70.7-72.2 GiB GTT at 8K-256K (Q2-1.0), i.e. it holds the whole file in GTT -> does not fit; halogen v2 62 GiB / ht43 ~54 GiB resident -> does not fit; ds4-server CUDA only; SGLang + DFlash2 needs the 173 GiB FP8 or 330 GiB BF16 checkpoint -> no. Conclusion: only formats that keep the backbone <= ~36 GiB resident fit, which is exactly prod’s GSQ-RCO Q2_0, REAP-288, Gyro-S TQ1_0 and Swift. Engine-side wins must therefore come from EngramHalo / strix-llama / TheRock / Vulkan-perf running the same GGUF family. Revisit if GTT or RAM grows.

Guide for host config (https://github.com/hogeheer499-commits/strix-halo-guide, evidence reviewed 2026-10-01; first-party machine is a 128 GB Beelink GTR9 Pro, so absolute numbers do not transfer to metis’ 64 GB): BIOS UMA carve-out 512 MB, ttm.pages_limit (+ amdgpu.gttsize, deprecated in kernel master in favour of ttm.pages_limit), amdgpu.cwsr_enable=0 as the gfx1151 MES-hang workaround (ROCm#5590) - metis does not set cwsr_enable; AMD SMI 27.0.0 offers amd-smi set --mem-carveout / --gtt instead of hand-editing modprobe + initramfs (needs Linux 7.0+, VBIOS ATCS 0xA + IGP info table v2.3, reboot). Remove AMDVLK so RADV is the only Vulkan ICD.

Source: Wiki.js