Qwen/Qwen3.6-35B-A3B-FP8

Checks that passed

• Greedy parity, 32 prompts of 5K to 31K tokens, cache hit vs cold: 32/32 identical with bit-identical logprobs; cold vs cold 32/32
• Cross-request isolation, 24 rounds: 0 leaks; images served from the cache, 8 cases: hit equals cold 8/8
• Needles at 65K, 130K and 250K, three depths: 9/9 cold, 9/9 from the cache, hit equals cold 9/9
• 16 images per prompt: 16/16 colours named in order (thinking off), a 17th rejected with a clean 400
• Requests sharing a prefix in the same step, 24: 24/24 correct; 11/24 worded differently from sequential runs, including 2 requests that had no cache hit. With prefix caching off the same test words 16/24 differently: batch composition on this MoE, not the cache
• Forced tool calls (tool_choice required): whitespace runaway 0/32
• Tool reliability, 3 cases x 40 sampled: 120/120 with strict think-end
• prompt_logprobs: aligned, in vocabulary and consistent with decode logprobs (plp check PASS)
• Agent replay, 4 sessions 8K to 64K: wall 130.5 s vs 607.5 s with caching off, later-turn TTFT p50 3.7 vs 18.3 s, 78% of prompt tokens served from the cache; 0 preemptions
• Functional set (arithmetic, factual, code, multi-turn, streaming, tool calls): 10/10 (earlier run, prefix caching off)
• Decode 74.5 tokens/s single stream, 204.8 tokens/s per card at 8 in flight, with fp8-moe-gather (earlier run, prefix caching off)
• Gathered-expert MoE vs the stock op: no element beyond 2 FP8 ULPs

Notes

• Prefix caching needs gdn-prefix-cache: on the stock release, prefix-cache hits on Gated DeltaNet layers resume from the wrong state. The fix stores GDN state checkpoints in the KV pages of the cache and keeps the compact GDN slots (VLLM_COMPACT_GDN=1).
• With prefix caching on, the hybrid KV block is 1,152 tokens and prefill chunks end on block boundaries, so a 1,280-token batch carries one block of prompt. Prompts shorter than one block never hit.
• --max-num-seqs and VLLM_HPU_MOE_GATHER_RATIO move together: 9+ decode tokens fall back to the stock MoE op at about 19 tok/s per stream (16 sequences would need a ratio of 0.5, not measured).
• Loading under a CPU quota can take 78 minutes: vllm-gaudi sizes its load thread pool from the host's core count, and the per-tensor FP8 rescale is then CPU-throttled. Run without a CPU limit, or cap the pool at the quota.

Qwen's 35B mixture-of-experts hybrid model (3B active, 256 experts, top 8; Gated DeltaNet plus attention) with vision and reasoning, from the official FP8 checkpoint, at its native 256K context on a single Gaudi 2 card, with prefix caching on through the gdn-prefix-cache fix.

The checkpoint declares the same architecture class as Qwen3.5-35B-A3B-FP8 (Qwen3_5MoeForConditionalGeneration), which serves with the same recipe.

Prefix caching

On the stock release, prefix-cache hits on the Gated DeltaNet layers resume from the wrong state, so prefix caching is only correct with gdn-prefix-cache, which stores GDN state checkpoints in the attention pages of the KV blocks the GDN groups own and keeps the compact GDN slots. See Qwen3.6-27B-FP8 and Prefix caching on Gated DeltaNet hybrids for the design.

On this model the hybrid KV block is 1,152 tokens; prefill chunks end on block boundaries, so each 1,280-token batch carries one block, and prompts shorter than 1,152 tokens never hit. The start-up log:

GPU KV cache size: 1,581,826 tokens
[gaudi-fix:gdn-prefix-cache] active: GDN groups [0, 1, 2], block 1152 tokens, 1412 KV blocks, 10 layer positions; ...
[gaudi-fix:gdn-prefix-cache] scheduler: GDN prefill chunks end on 1152-token blocks

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0 (see Using a recipe). The reasons are the same as on Qwen3.6-27B-FP8 (including prompt-order, resumed-prefill-logits, hybrid-decode-pad and the opt-in gdn-prefix-cache), without structured-outputs-auto-whitespace, plus:

This recipe does not set disable_any_whitespace: forced-tool whitespace runaway measured 0/32 here without it (prefix caching on).

Verified

On vllm-gaudi v0.26.0, one Gaudi 2 card, the launch above as written (prefix caching on with gdn-prefix-cache):

• Greedy parity, 32 prompts of 5K to 31K tokens (single questions, multi-turn agent sessions with tools, extensions, diverging prefixes; thinking on and off), the same request served from the cache and cold: 32/32 identical, all with bit-identical logprobs. Two cold runs: 32/32 identical.
• Cross-request isolation, 24 rounds with an unrelated request between fill and hit: 0 leaks.
• Images served from the cache, 8 cases: hit equals cold 8/8.
• Needles at 65K, 130K and 250K tokens, three depths each: 9/9 cold, 9/9 from the cache, hit equals cold 9/9.
• 16 images in one prompt: 16/16 colours named in order (thinking off), a 17th image rejected with a clean 400.
• Requests sharing a prefix in the same step, 24: 24/24 found the right code. 11 of 24 were worded differently from the same request sent alone, including 2 that had no cache hit: on this MoE the batch a request decodes in changes its greedy wording. With prefix caching off the same test words 16 of 24 differently, so the cache adds nothing to it.
• Forced tool calls (tool_choice required): whitespace runaway 0/32.
• Tool reliability, 3 cases x 40 sampled requests: 120/120 with strict think-end.
• prompt_logprobs: ids aligned with the prompt and in vocabulary, values consistent with decode logprobs (plp check PASS).
• Agent replay, 4 sessions growing from 8K to 64K tokens: 0 preemptions, 0 engine restarts across all of the above.

From earlier runs of this recipe with prefix caching off (the decode path is the same):

• Functional set (arithmetic, factual, code, lateral, multi-turn, streaming, tool calls): 10/10.
• Gathered-expert MoE against the stock op: no output element beyond 2 FP8 ULPs.
• Tool reliability 240/240 over two runs with strict think-end, 115/120 without.

Performance

Single measurements on one card, vllm-gaudi v0.26.0 with fp8-moe-gather (same gate as above), prefix caching off, every batch size warmed first, 256 output tokens per stream, ignore_eos, two repetitions.

The curve has not flattened at 8; see "What not to do" before raising the cap.

Needle requests on the launch above (prefix caching on), the same pod, each prompt sent cold and then again from the cache:

The cold TTFTs vary by depth and run order (first-touch graph compiles in compile mode); treat them as a range, not a curve.

Agent replay, 4 sessions growing from 8K to 64K tokens, 200 output tokens per turn, 4 in flight:

Known issues

• Prompts shorter than one 1,152-token block never hit the cache.
• Cold load on the test pod (no CPU limit, weights on a network file system) took about 19 minutes from start to a ready KV cache.
• Cold start under a CPU quota: loading took 78 minutes because vllm-gaudi sizes its load thread pool from the host's core count while every FP8 expert tensor (about 61,000) is rescaled on the CPU, and the pool is throttled by the quota (about 73 ms per tensor instead of 0.2 ms). With the pool capped at the quota the load took 46 to 49 s and the server was ready in under 2 minutes. Run without a CPU limit, or cap the pool.
• Weights from Hugging Face over Xet can stall silently for these repos; HF_HUB_DISABLE_XET=1 avoids it when pre-fetching.

What not to do

• Do not turn on prefix caching on this release without gdn-prefix-cache: hits then resume the GDN layers from a wrong state.
• Do not raise --max-num-seqs above 8 without raising VLLM_HPU_MOE_GATHER_RATIO with it (16 sequences: 0.5), and re-measure. A decode batch of 9 or more otherwise falls back to the stock fused op, and every stream in it drops toward about 19 tokens/s.
• Do not use lazy mode for this model.