niklas‑stephan.de

Gyro (TQ trellis) quants for Flash-Next: compatibility and fit on metis

Last updated 06/11/2026 · llm, qwen, metis, quantization, strix-llama

Verified 2026-11-01 against the actual repos and the live container.

Gyro (agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)

  • On-disk files (n-gram table embedded, stays on disk with —ngram-on-disk): Gyro-S TQ1_0 54.47 GiB (27.64 GiB GPU weights), Gyro-M TQ2_0 85.65 GiB (34.98 GiB), Gyro-L TQ3_0 100.88 GiB (50.21 GiB). Repo updated 2026-10-25, ~607k downloads.
  • Format: trellis-coded quant + Hadamard rotation (QTIP family), custom types TQK6/TQK7, TQ2_T, TQP5-7. Upstream llama.cpp cannot load them.
  • Runtime: halo-box/strix-llama.cpp default branch is MASTER (not main); its tree contains Vulkan tq shaders (dequant_tq1_0/tq2_0/tqk6/tqk7/tqp5-7, mul_mat_vec_tq*) and CUDA tq.cuh. Active commits 2026-10-31.
  • GOTCHA CORRECTED 2026-11-06: the 12263-era image had no TQ symbols, but current prod (strix-llama build 12297, commit ee30b949e) DOES: libllama.so contains TQ2_T/TQK6/trellis symbols and ggml.h defines TQ1_0/TQ2_0/TQ2_T/TQK6/TQK7/TQP5-7 - Gyro files load in prod today. Vulkan fused MUL_MAT_ID tile for TQ2_T (fork PR #252) is still open, so fused decode speed not yet realized.
  • Vendor-reported quality (self-reported): Gyro-M KLD 0.291/top-1 78.75% in 34.98 GiB vs ISTA IQ3_XXS 0.311/77.68% in 43.80 GiB; Gyro-S 0.401/75.29% in 27.64 GiB vs IQ2_XS 0.420/74.22% in 36.52 GiB. Decode on Strix Halo (strix-llama Vulkan): Gyro-S 36.1, Gyro-M 30.1 tok/s - slower than prod GSQ-RCO Q2_0 MTP (43-50 tok/s). HIP with MTP: 57/58/54 tok/s on code.
  • Fit vs metis ceiling: Gyro-S/M fit (35 GiB weights + MTP draft 2.51 GiB + KV/compute); Gyro-L (50.21 GiB) exceeds the practical 45-50 GiB ceiling with KV/mmproj.
  • Disk: 54-86 GiB downloads; ZFS root 647 GiB - keep one variant plus the existing Q2_0 until an A/B decides.

ISTA-DASLab GSQ-RCO Coder (expert-pruned)

  • 256 of 512 experts removed per layer, retained weights 3.5 bpw; IQ1_M shards 27.58 + 26.82 GiB; n-gram shard may stay on disk. README update 2026-10-17: SWE-bench Verified 79.20, LiveCodeBench v6 87.43. Code-focused; degradation outside code/agent/vision is accepted cost - general agent stack should use the unpruned GSQ-RCO line. Standard GGUF types - loadable by the current engine.

pugant ROCmFP4_STRIX_LEAN

  • 57.04 GB, ~55 GiB peak host: exceeds metis’ 56 GiB GTT ttm limit with KV/mmproj, and author documents ~2.17x perplexity vs reference. Skip.

n-gram question (answered)

metis already runs -lm mmap —lazy-mode on-direct (llama.cpp >=12xxx: —ngram-on-disk is the deprecated alias for —lazy-mode on); the 28.8 GB shard is disk-resident via page cache. No memory win there; page-cache starvation is the known pressure (see llm-memory-management-flashnext).

If testing Gyro

Cannot run beside prod (~44-48 GiB GTT each). Plan: stop llama-flashnext, run Gyro-S/M on the test variant (port 8130) with a strix-llama master build or agention container, then swap back per llm-serving-metis rollback.

TODO (open, 2026-11-01 - not started, Niklas deferred)

  1. Download Gyro-S TQ1_0 (54.47 GiB) from agentionai/Qwen3.8-Flash-Next-Gyro-GGUF into /data/docker/llama/models/ (keep Q2_0 baseline; do NOT also grab Gyro-M/L until the A/B decides).
  2. Run it on the test variant (port 8130, variants/test-8130) with prod down; read the startup log’s device-assignment lines to check whether TQ/trellis MoE layers land on ROCm0/HIP or fall back to CPU (the decisive risk - CPU kernels exist, HIP kernels unconfirmed).
  3. If GPU-resident: A/B Gyro-M vs current GSQ-RCO Q2_0 on identical prompts; measure GTT, RssAnon, decode t/s, cold prefill, loop behavior. Vendor HIP+MTP target: 57-58 tok/s on code.
  4. Only if it beats baseline, move to prod and update llm-serving-metis.

Source: Wiki.js