Memory

Memory model on unified-memory systems, size of the quantizations, behavior of the embedding table per engine, and how the memory requirement is calculated.

Memory split on Strix Halo

A Strix Halo has one memory shared by CPU and GPU. On the test machine the operating system sees 109.7 GB; of that the GPU gets 16 GB fixed (VRAM) and may use the rest as GTT. Everything shares the same physical RAM: weights, KV cache, compute buffers, the embedding table, the page cache and the operating system.

Quantizations

All four contain the same per_layer_token_embd table: 160 × 320 million entries, IQ4_NL, 26.8 GB. The only difference between the quants is the precision of the routed experts. (GB here means GiB; unsloth states decimal GB.)

Embedding table per engine and load mode

Cause of the long mmap load time on the stock fork: the weights are read page by page via page faults, without readahead, and the page cache competes with the GTT for the same RAM. As soon as the model no longer fits into memory twice, pages are evicted and read again: 149 GB of reads for a 104 GB file. EngramHalo issues readahead hints and frees the page cache behind the upload.

Memory estimate in the program

• Weights in the GPU (file minus table) plus 2.7 GB of table with EngramHalo, otherwise plus 26.8 GB.
• KV cache of the 12 full-attention layers: 2 KV heads × 256 dimensions → 24 KB per token at f16, 12.75 KB at q8_0; plus indexer cache (K 128 + V 256 dimensions). Measured at 32k q8_0: 408 + 153 MB.
• Recurrent DeltaNet state: 113 MB per slot, independent of the context.
• Compute buffer: 297 MB at ubatch 512, 1188 MB at ubatch 2048 (measured), plus host buffer.
• MTP draft model 2.1 GB plus its own compute and KV buffer.
• Prompt cache (--cache-ram) as the upper bound; 6 GB reserve for operating system and page cache.

Example, default preset (EngramHalo, Q4_K_XL, 128k, MTP): estimated 90.0 GB, measured 89.7 GB.

Context length

Because only 12 of the 48 layers have a KV cache, 256k of context with q8_0 is only 3.3 GB of KV plus 0.8 GB of indexer. With this model the context length is above all a question of prompt-processing speed, not of memory.

Concurrent contexts

With -np N the server serves N requests at the same time. The value of -c is then the total context across all slots: with -np 4 -c 262144 each slot gets 65,536 tokens. Two quantities determine what fits:

• KV and indexer cache: 17,952 bytes per token, that is 0.55 GB per 32k of context – independent of the quant, because only 12 of the 48 layers have attention and the KV type is q8_0.
• DeltaNet state: 113 MB per slot, independent of the context length. With 16 slots that is 1.8 GB.

The context of a single slot is limited to 262,144 tokens by the training length of the model, not by memory: these 256k fit with every quant, including UD-Q4_K_XL (95.5 GB used, 5.0 GB headroom). Only with several slots does memory become an issue.

Maximum number of concurrent contexts of the given size, with the Q8_0 MTP draft head (3.85 GB, the head all presets use), prompt cache 2 GB, ubatch 2048, 6 GB reserve for the operating system, based on 106.5 GB of free memory. Without MTP it is about 10 to 30 percent more, because the draft model and its KV cost around 4.7 GB. Against the smaller Q4_K_M head the difference is at most one slot, and at 256k none at all. The values are cut off at 64 slots; above that the number is theoretical anyway. Recalculate with bench/context_limits.py.

Slots and context per slot trade one for one. Once weights and draft head are resident, what is left is a fixed token budget; only the DeltaNet state depends on how it is split. For UD-IQ3_XXS with 4 GB headroom the total is around 1.9 M tokens however it is cut — 8 × 242k, 12 × 159k, 16 × 117k, 24 × 76k, 32 × 55k — so twice the users means half the context each. UD-IQ4_XS has about 1.33 M tokens to give away (6 × 224k, 8 × 166k, 16 × 80k). Choosing the number of users therefore already chooses the context length.

Context checkpoints: the cache of this model, and the largest hidden cost

llama.cpp keeps rollback points per slot (--ctx-checkpoints, default 32). For a hybrid recurrent model they are not a convenience but the only way a conversation can be continued from the cache at all. Switch them off and the server says so plainly: forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory).

One checkpoint is enough — 1 and 32 deliver the same 99.5 %, the rest is paid for nothing. And giving every session its own slot does not remove the need: even with nothing evicted the hit rate is 0.0 % without checkpoints, because a new turn has to drop the trailing token of the previous answer, and rolling the recurrent state back by a single token already needs one. What checkpoints do not buy is the repair of an edited history: rewriting the middle of a conversation costs a full re-processing with 4 checkpoints just as it does with none.

The slope is exact across independent runs (122.5 MB at 5038 tokens, 165.3 MB at 26688, 248.0 MB at 68539 in a Terminal-Bench run): the DeltaNet state is the floor, and what grows is the draft KV of the whole prefix. With MTP a checkpoint at 224k tokens is 555 MB, and the server keeps up to 32 of them per slot. That is why the table above assumes --ctx-checkpoints 0; with the default of 32, UD-IQ3_XXS at 256k drops from 8 concurrent contexts to 2 and UD-Q4_K_XL no longer fits even once. Keep the number small rather than switching it off.

Split or shared KV cache

With --no-kv-unified — the default as soon as -np is given — every slot owns a fixed n_ctx / n_parallel slice. With --kv-unified all slots draw from one pool and a single session may use up to the training length. Measured: a 26,692-token prompt against four slots of 16,384 is rejected with request exceeds the available context size; the same request against the same pool with a shared cache runs through at 34.1 tokens per second. The fork also treats idle slots differently — with a shared pool an idle slot is saved to the prompt cache and cleared, because that frees reusable room; with a split cache it is only copied and its KV stays in place. One slot per agent with a split cache is therefore the configuration in which nothing is ever evicted; a shared pool buys elasticity and pays with eviction and restore.

What works in practice

With the small quants memory is not the limit – throughput is. Measured with UD-IQ4_XS without MTP, 20,480 tokens per slot:

Total throughput still rises up to 8 slots, but the individual response keeps getting slower: with 8 concurrent users, 6.8 tokens per second per request remain. That is usable for chat, but not for agents – an agent that produces 30,000 tokens then waits 74 minutes instead of 25. That is why the agent benchmarks run with one slot.

Two further limitations: speculative decoding (MTP) only helps with one slot – with 8 slots throughput drops from 50 to 35 t/s. And on gfx1151 several slots need the patch from llama.cpp #25992, otherwise the server returns faulty responses from the second slot onwards.

Measured with long prompts (2026-09-05)

The numbers above come from a test with short prompts. With prompts typical of agents it looks different: one text of its own per user of around 16,000 tokens, 2,000 tokens of response, MTP on, UD-IQ4_XS.

Parallelism gains nothing here. The total throughput of prompt and output tokens lies at every level between 203 and 240 tokens per second – eight slots process just as much in the same time as one. With short prompts throughput rises with the number of slots, because the machine idles between tokens; with long prompts it is already fully occupied by a single stream, and more slots merely spread the same performance over more waiting requests. The individual response slows from 29.9 to 2.8 tokens per second in the process.

On top of that, the output breaks at eight slots. Every user was to begin their response with a prescribed code word. With one to four slots that always works; with eight only two out of eight manage it – two responses stay empty, one repeats the prompt, three corrupt the code word by one or two characters. No slot ever gets to see another slot's code word, so there is no cross-talk between the slots; the damage arises within the individual response. That matches the description in llama.cpp issue #27572, where with several slots and long prompts garbage gets into the context of the MTP head.

Consequence for operation: for work with long context – agents, code analysis, long documents – one slot. Several slots are only worth it with short prompts, that is chat. The fix for the output problem is available as engine/patches/0003-27311-uma-ring-buffer.patch, but has not been built: it would fix the broken responses, but not the missing throughput gain – and that is the actual reason to run one slot with long context.