Decisions

Configuration decisions with rationale, evidence and rejected alternatives.

Backend: ROCm/HIP instead of Vulkan

Decision: ROCm/HIP (gfx1151), both engines as a HIP build.
Rationale: Community measurements often show Vulkan 10–25 % ahead at plain decoding and ROCm 20 % ahead at prompt processing. With MTP, however, Vulkan collapses on this architecture (21 → 5 tokens/s), because the recurrent state is written to host memory on every speculation round. The machine also lacks the Vulkan development packages.
Evidence: llama.cpp PR #27836 (Vulkan regression with MTP), soothill.io Vulkan vs. ROCm comparison; own Vulkan build failed on missing headers.
Rejected: Vulkan build of the patched sources (prepared in engine/build.sh, not measured).

Engine: EngramHalo.cpp (default) and llama.cpp with MTP patch

Decision: EngramHalo.cpp (branch strix-halo-qwen4exp) is preferred; the stock fork remains as a fallback.
Rationale: EngramHalo keeps the 26.8 GB embedding table lazy in every load mode (measured 2.7 GB in RAM instead of 28 GB), loads in 16–28 seconds, has a HIP top-k kernel against the CPU fallback from 1k context on, and a sparse QSA gather from 16k context on. That makes UD-Q4_K_XL fit with MTP; with the stock fork it does not fit.
Evidence: Own footprint measurements (bench/results/mem): Q4_K_XL + MTP 84.7 GB on EngramHalo, kernel OOM on the stock fork. Fork documentation: docs/strix-halo/README.md.
Rejected: Stock fork with --load-mode mmap: table lazy, but weight upload at 18 MB/s (Q4_K_XL: 140 minutes).

Quantization: UD-Q4_K_XL (alternatives IQ4_XS, IQ3_XXS)

Decision: Default preset UD-Q4_K_XL (KLD 0.047, top-1 92 %). IQ4_XS for less memory, IQ3_XXS for maximum speed at small context.
Rationale: The quants differ only in the precision of the routed experts; attention, DeltaNet and the embedding table are the same everywhere. At 35 tokens/s Q4_K_XL is practically as fast as IQ4_XS (36.5) and Q2_K_XL (35), so there is no reason to sacrifice quality once it fits.
Evidence: unsloth KLD table; own measurements with all four quants.
Rejected: Q2_K_XL as the default: 17 % faster without MTP, but equally fast with MTP and clearly worse (KLD 0.225).

MTP head: dzannotti, Q8_0 for deep context

Decision: Only heads with dzannotti's tensor layout are used; the program recognizes compatibility by the tensor names. All presets ask for a Q8_0 head quantized from the published BF16 head, falling back to the 2.4 GB Q4_K_M head when it is absent.
Rationale: The local builds follow the graph from llama.cpp PR #27739 and expect the tensors output_hc_*. The unsloth head names them blk.48.nextn.hc_head_* (PR #28243) and loads only with the unsloth fork.
Evidence: GGUF headers of both heads read; loader code in src/models/qwen4exp.cpp of both forks.
Draft head size: Measured with UD-IQ4_XS, one slot, 600 output tokens: Q4_K_M reaches 40.9 t/s at an empty context but only 26.8 t/s at 30k, where its draft acceptance falls from 0.90 to 0.62. The Q8_0 head (3.85 GiB, quantized here) holds acceptance at 0.83 and reaches 29.0 t/s at 30k, while giving up 4.4 t/s at an empty context. BF16 (7.24 GiB) is slowest at every depth. At 64k the two heads meet again at 20.0 to 20.5 t/s: attention over that much context dominates the step, so the better acceptance no longer buys speed.
Rejected: unsloth head with the unsloth fork b10715-mix: not tested. BF16 head as the default: bigger and slower, the draft step is dominated by the output projection.

MTP: enabled unless several requests really run at once

Decision: All presets use --spec-type draft-mtp,ngram-mod --spec-draft-n-max 4 --spec-draft-p-min 0.75, including the multi-agent preset; only sustained simultaneous load is a reason to switch it off.
Rationale: One user: 20.8 → 33–41 tokens/s. Eight users generating at the same time: 35 tokens/s in total with MTP, 50 without. Agents are bursty and usually compute alone, so the draft head keeps paying there; what costs is eight slots under continuous load. Between n_max 3 to 6 and p_min 0 to 0.75 there is only about 5 % difference (measurement noise 3 %). Temperature 1.0 costs only ~5 % acceptance compared to greedy.
Evidence: MTP fine-tuning and multi-user sweep in bench/results.
Rejected: n_max 16 / p_min 0.8 (recommendation for the RTX 5090): does not transfer to bandwidth-limited hardware.

Several agents on one server: one slot each, and exactly one checkpoint

Decision: Preset eh-team: UD-IQ3_XXS, -np 8 -c 1048576 (8 × 128k), split KV cache, --ctx-checkpoints 1, --cache-ram 2048, MTP on.
Rationale: Context checkpoints are not a convenience on this model, they are the cache: with --ctx-checkpoints 0 every follow-up turn is re-processed in full — 0.0 % cache hit instead of 99.5 %, 20.5 s instead of 4.2 s — even when each session has its own slot, because a new turn has to roll the recurrent state back by the trailing token of the previous answer. One checkpoint delivers exactly the same hit rate as the default of 32, and the other 31 only cost memory: 112.6 MB each plus 2072 bytes per token of prefix. A split cache pins each agent to its slot so nothing is ever evicted.
Verified: Eight sessions on eight slots hit the cache to 99.5 % from the second turn on (4–5 s per turn instead of 20 s), a single session with a 53,511-token prompt to 99.9 % (5.7 s instead of about 200 s), with 14 GB of memory still free. Beyond the slot count the RAM prompt cache decides: at --cache-ram 4096 it holds ten entries of 378 MB, and twelve sessions on eight slots fall back to a 0 % hit rate.
Evidence: Four measurement series with bench/cache_probe.py (results in bench/results/cache/), server log messages forcing full prompt re-processing due to lack of cache data and [TAG_IDLE_SLOT_CLEAR] in the fork.
Sizing: Users and context trade almost one for one, but each slot also costs about 750 MB of compute buffer — measured from the server's own memory breakdown, 1188 MB at one slot against 6442 MB at eight. Two attempts (8 × 196k and 8 × 160k) were killed by the memory guard; both terms are now in the estimator, which is nonetheless still optimistic by several GB at eight slots — treat its headroom as an upper bound. 6 × 176k and 4 × 256k fit just as well, 16 slots do not. Plan with bench/serving_plan.py.
Rejected: --kv-unified: lets one session use the whole pool (a 26,692-token prompt that a split cache rejects runs through), but idle slots are cleared from the pool and have to be restored. --ctx-checkpoints 0: saves the most memory and destroys the cache. --cache-reuse: not switched off automatically for this model, but the recurrent state cannot be shifted like attention KV, so it stays at 0 and untested.

KV cache: q8_0 with Hadamard rotation active

Decision: -ctk q8_0 -ctv q8_0, LLAMA_ATTN_ROT_DISABLE stays unset.
Rationale: q8_0 halves the KV memory at practically the same quality (KLD 0.0018) and the same speed. The environment variable comes from a pre-release version of the model support; the current code rotates Q/K/V itself. Without the variable the model loads, and the rotation costs only 2 %.
Evidence: Load test with rotation (attn_rot_k = 1): 24.2 instead of 24.6 tokens/s. llama.cpp PR #21038.
Rejected: f16 KV: slower at depth; q4_0 KV: destroys quality (KLD 5.5).

Load mode: auto (stock fork) or none (EngramHalo)

Decision: Stock fork: --load-mode auto. EngramHalo: -lm none; the table stays lazy anyway.
Rationale: ROCm reports “no mmap”; with auto, llama.cpp copies the table into RAM. Forced mmap makes the table lazy, but loads the weights page by page without readahead: 18 MB/s, 140 minutes for Q4_K_XL. EngramHalo has readahead and drop-behind and loads in 39 seconds even with mmap.
Evidence: Own measurements q4-mmap (stock, 140 min) and eh-q4kxl-mmap-mtp (39 s).

Thinking mode and sampling

Decision: reasoning_effort “medium” for everyday use, “xhigh” (template default) only when needed; thinking sampling temp 1.0 / top_p 0.95 / top_k 20, non-thinking temp 0.7 / top_p 0.8 / presence 1.5.
Rationale: The chat template accepts exactly xhigh, medium, low and throws an exception on anything else. The sampling values are in the model card and in the GGUF.
Evidence: Chat template read from the GGUF; Qwen model card; unsloth documentation.

Multi-user: workaround for mixed-up responses (#25992)

Decision: The workaround for llama.cpp #25992 (no pinned host buffers on integrated GPUs) is built into both builds.
Rationale: On gfx1151, without the patch, responses were swapped between slots. The multi-user benchmark checks this with a code word per user: not a single mix-up in 30 concurrent requests.
Evidence: EngramHalo Dockerfile.rocm-7.14 and patch file; own multi-user sweep.

Memory check and guard

Decision: Before the start the program computes the requirement against MemAvailable; during operation a guard stops the server before the kernel OOM killer strikes.
Rationale: GPU memory (GTT) does not show up in the RSS of the process; only MemAvailable shows how close an OOM is. Two kernel OOM kills during work took the whole session down with them.
Evidence: Kernel log 2026-09-03 12:56 and 13:03; footprint measurement series with memguard.