Overview

Documentation on running Qwen3.8-Flash-Next on AMD Strix Halo (128 GB unified memory): configuration, memory model, measurements and sources – for llama.cpp and for the second engine, halogen-flash-server.

Introduction

Qwen3.8-Flash-Next is a large mixture-of-experts model (125 billion parameters, 6 billion active) with one peculiarity: a 51-billion-parameter table for n-gram embeddings. On a Strix Halo machine with 128 GB of unified memory the model only fits if engine, quantization, load mode and context fit together. This page documents which decisions were made, why, and what was measured.

The work started with llama.cpp and stayed there for most of these pages. Since September 2026 a second engine runs on the same machine: halogen-flash-server, written for this chip alone. Both servers claim most of the memory, so they take turns, and one endpoint sits in front of them – the client router never learns which of the two answered.

The accompanying tool is a terminal program (Python/Textual) that takes care of the configuration, computes the memory requirement in advance, starts and monitors the server and runs benchmarks. See Guide.

Summary

1. Memory: The n-gram table is 26.8 GB in every quant. On ROCm, standard llama.cpp loads it into RAM in addition to the GPU weights. Memory
2. Engine: EngramHalo.cpp (Strix Halo fork) leaves the table on the NVMe, loads in under 30 seconds and makes the best quant usable with MTP. Decision
3. MTP: Speculative decoding raises throughput for a single user from 21 to 33–41 tokens/s; from two concurrent users on it lowers total throughput. Measurements
4. MTP head: The dzannotti head has the tensor names the builds expect; the unsloth head requires a different fork. Decision
5. Backend: ROCm/HIP. Vulkan is often faster at plain decoding, but loses clearly with MTP on this model. Decision
6. Second engine: halogen-flash-server is not a llama.cpp fork but its own HIP engine for gfx1151, with its own 5.53-bit checkpoint. It is ahead everywhere the wait is long: prefill, several users at once, and resuming a conversation. Below
7. Thinking harder does not pay: the same benchmark at reasoning effort xhigh solved fewer tasks than medium and took 3.3 times as long. Terminal-Bench

Decode speed (single user)

EngramHalo.cpp, 32k context, KV cache q8_0, dzannotti head. All numbers: Measurements.

Second engine: halogen-flash-server

halogen-flash-server serves the same model a second way: not a llama.cpp fork but its own HIP engine written for gfx1151 only, shipped as a container with its own 5.53-bit checkpoint, a quality overlay, an MTP head and a vision sidecar. It has no authentication of its own and keeps its prompt cache on disk, so a conversation survives a restart.

Measured through the one endpoint – keys, HTTP, streaming – against the running service, both on the same day; the single-user row for halogen comes from image 0.5.6, the others from 0.10.2. halogen wins wherever the wait is long; the llama.cpp service holds more long conversations at once.

Terminal-Bench-Mini-20

On the agent benchmark halogen solved 18 of 20 tasks on the first attempt and all 20 with a single retry, in 3 h 46 min. The llama.cpp quants solved 15 to 16 on the first attempt and needed 6 h 27 min to 8 h 16 min for the run. Across all tasks halogen generated 39.4 tokens per second against 16.8 to 21.9. Raising the reasoning effort to xhigh made it worse, not better: 17 of 20, 18 with a retry, 12 h 30 min and 1.1 million output tokens instead of 353,000. Every task, both attempts and the agent transcripts are on the Terminal-Bench page.

One endpoint for both

Each server claims most of the machine, so only one runs at a time. A small router on port 8080 checks the API keys, finds whichever server is up, waits during a switch instead of failing, and fills in the same defaults for whatever a request leaves open – so the client router (LiteLLM) needs a single entry and never learns which engine answered. Details: halogen/ and router/.