Qwen/Qwen3.6-27B-FP8

27B dense hybrid (Gated DeltaNet + attention) reasoning model with vision, FP8, native 256K context on one Gaudi 2 card.

Verified on Gaudi 2 Hugging Face Full guide

dense27B262,144 ctx servedfp8vllm-gaudi v0.26.0Multimodal

Hardware

1 card · TP 1

Image

vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1

Variant

The official FP8 checkpoint (block 128 x 128, dynamic activations), one Gaudi 2 card.

Features

Fixes

profile v0.26.0 gdn-prefix-cache qwen3-strict-think-end

15 fixes this configuration depends on; opt-in fixes are named.

Launch

Shell

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3.6-27B-FP8", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "max_tokens": 512}'

export PT_HPU_LAZY_MODE=0
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_USE_NAIVE_MAMBA_CACHE_SHARING=1
export VLLM_COMPACT_GDN=1
export VLLM_QWEN3_STRICT_THINK_END=1
export PT_HPUGRAPH_DISABLE_TENSOR_CACHE=false
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.3
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=8
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=32
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=4096
export VLLM_PROMPT_SEQ_BUCKET_MAX=262144
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=16384
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile plus this recipe's opt-in fixes, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 \
    gdn-prefix-cache qwen3-strict-think-end || exit 1

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.85 \
  --block-size 128 \
  --max-num-batched-tokens 1792 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --structured-outputs-config '{"disable_any_whitespace": true}' \
  --enable-prompt-tokens-details \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image":16,"video":0}'

Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.

At a glance

Checks that passed

Greedy parity, 32 prompts of 5K to 31K tokens, cache hit vs cold: 31/32 identical, the other forked at an exact tie (both top-2 logprobs -0.6932); cold vs cold 32/32
Cold greedy outputs, caching on vs off: 27/32 identical, inside the noise floor of changing the prefill chunk (caching off at 1,792 vs 896 batched tokens: 20/32)
Cross-request isolation, 24 rounds: 0 leaks; images served from the cache, 8 cases: hit equals cold 8/8
Requests sharing a prefix in the same step, 24: 24/24 correct; 22/24 word for word as when sent one at a time (batch-composition wording, one of them without a hit)
Needles at 65K, 130K and 250K, three depths: 9/9 cold, 9/9 from the cache, hit equals cold 9/9
16 images per prompt: 16/16 colours named in order (thinking off), a 17th rejected with a clean 400
Forced tool calls (tool_choice required): whitespace runaway 0/32 with disable_any_whitespace
Tool reliability, 3 cases x 40 sampled at 0.6: 119/120 with strict think-end
prompt_logprobs: aligned, in vocabulary and consistent with decode logprobs (plp check PASS)
Agent replay, 4 sessions 8K to 64K: wall 171 s vs 513 s with caching off, later-turn TTFT p50 6.7 vs 24.2 s; 0 preemptions, 0 restarts

Notes

Prefix caching needs gdn-prefix-cache: on the stock release, prefix-cache hits on Gated DeltaNet layers resume from the wrong state. The fix stores GDN state checkpoints in the KV pages of the cache and keeps the compact GDN slots (VLLM_COMPACT_GDN=1).
With prefix caching on, the hybrid KV block is 896 tokens and prefill chunks end on block boundaries, so a 1,792-token batch is two blocks. Prompts shorter than one block never hit.
Qwen3.5-27B-FP8 has the same geometry and runs this recipe unchanged (drop VLLM_QWEN3_STRICT_THINK_END and its fix: it scores 120/120 without).

Guide

Qwen's 27B dense hybrid model (48 Gated DeltaNet layers and 16 attention layers) with vision and reasoning, from the official FP8 checkpoint, at its native 256K context on a single Gaudi 2 card, with prefix caching on through the gdn-prefix-cache fix.


Model	Qwen/Qwen3.6-27B-FP8
Hardware	1x Gaudi 2 (TP 1)
Image	vllm-gaudi `v0.26.0` (`b059f9c`), vLLM `568afb3a`, Gaudi SW 1.24.1
Precision	FP8 checkpoint (block 128 x 128, dynamic activations)
Context	262,144 (native)
Concurrency	`--max-num-seqs 32`
Parsers	tools `qwen3_coder`, reasoning `qwen3`
Recipe	`recipes/Qwen/Qwen3.6-27B-FP8/recipe.yaml`

Qwen/Qwen3.5-27B-FP8 has the same geometry and runs this recipe unchanged, except that it does not need qwen3-strict-think-end (120/120 on tool reliability without it). Some measurements below come from it and are marked.

Prefix caching

On the stock release, prefix-cache hits on the Gated DeltaNet layers resume from the wrong state, so prefix caching is only correct with gdn-prefix-cache. The fix keeps the GDN layers in their compact per-request slots (VLLM_COMPACT_GDN=1) and stores a state checkpoint at the end of every full KV block in that block's own attention page, which a block owned by a GDN group does not otherwise use. A hit restores the checkpoint, bit for bit, before the next chunk. The full KV pool stays available: 623,359 tokens at the flags below.

With caching on, the hybrid KV block is 896 tokens (one page holds a whole GDN state) and prefill chunks end on block boundaries. The start-up log confirms both:

[gaudi-fix:gdn-prefix-cache] active: GDN groups [0, 1, 2], block 896 tokens, 711 KV blocks, 16 layer positions; ...
[gaudi-fix:gdn-prefix-cache] scheduler: GDN prefill chunks end on 896-token blocks

The design, the earlier measurements and the upstream status are in Prefix caching on Gated DeltaNet hybrids.

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe). The profile also runs seeded-sampling; this recipe does not depend on it (the seed segfault is a lazy-mode bug).

Fix	Why this model needs it
`prompt-order`	with prefix caching, a request that hits a block another request writes in the same step read it before it was written
`sampling-rows-refresh`	stale device sampling rows after a batch reorder
`structured-output-rows`	structured-output steps sample each row with another request's parameters
`bad-words-index`	`bad_words` raises `IndexError` in the engine core
`logitsprocs-rows`	`logit_bias`, `min_p` and `min_tokens` are silently ignored
`async-discard-counter`	`AttributeError` on the first preemption under async scheduling
`resumed-prefill-logits`	a request resumed after preemption with prefix-cache hits samples again at every recomputed position, so the generated tail repeats
`hybrid-decode-pad`	when the hybrid KV block is split into 128-token kernel blocks, decode padding writes into a live block of another request; the start-up log says whether the pad was moved
`gdn-prefix-cache` (opt-in)	prefix-cache hits on the GDN layers resume from a wrong state on the stock release; the fix stores and restores GDN checkpoints in the KV pages and zeroes new blocks and slots. It needs `hybrid-decode-pad` first
`gather-logprobs-int32`	the id corruption is lazy-mode only, but the fix is harmless here and keeps one fix set across recipes
`prompt-logprobs`	multimodal model with one prompt in a prefill batch returns flat hidden states; the first `prompt_logprobs` request kills the engine
`logprobs-decode-guard`	a bad logprob id from any source is replaced instead of stopping the API server
`rope-cpu-cache`	Gaudi fp32 cos/sin are wrong past about 1.5e5, so RoPE tables built on the HPU are wrong from position about 196K
`structured-outputs-auto-whitespace`	lets `disable_any_whitespace` work with the `auto` backend, which keeps the fallback to guidance for schemas xgrammar cannot compile
`qwen3-strict-think-end` (opt-in)	v0.26.0's Qwen3 parser ends reasoning at a `<tool_call>` the model quotes while thinking (the template's example) and returns it as a real call; with `VLLM_QWEN3_STRICT_THINK_END=1` only `</think>` ends reasoning

Verified

On vllm-gaudi v0.26.0, one Gaudi 2 card, the launch above as written (prefix caching on with gdn-prefix-cache):

Greedy parity, 32 prompts of 5K to 31K tokens (single questions, multi-turn agent sessions with tools, extensions, diverging prefixes; thinking on and off), the same request served from the cache and cold: 31/32 identical, 27/32 with bit-identical logprobs. The other one forked at an exact tie (the two top tokens both at logprob -0.6932). Two cold runs: 32/32 identical.
Cross-request isolation, 24 rounds with an unrelated request between fill and hit: 0 leaks.
Requests sharing a prefix in the same step, 24: 24/24 correct; 22/24 word for word as when sent one at a time (the two differences are batch-composition wording, one of them on a request without a cache hit).
Images served from the cache, 8 cases: hit equals cold 8/8.
Needles at 65K, 130K and 250K tokens, three depths each: 9/9 cold, 9/9 from the cache, hit equals cold 9/9.
16 images in one prompt: 16/16 colours named in order (thinking off), a 17th image rejected with a clean 400.
Forced tool calls (tool_choice: required): whitespace runaway 0/32 with disable_any_whitespace.
Tool reliability, 3 cases x 40 sampled at 0.6: 119/120 with strict think-end (one wrong tool).
prompt_logprobs: ids aligned with the prompt and in vocabulary at every size checked, values consistent with decode logprobs (median difference 0.001 to 0.007).
Agent replay, 4 sessions growing from 8K to 64K tokens: 0 preemptions, 0 engine restarts across all of the above.

Performance

Single runs on one card, vllm-gaudi v0.26.0, warm.

Test	Prefix caching on (this recipe)	Prefix caching off	Measured as
Agent replay, 4 sessions 8K to 64K, 200 output tokens per turn: wall	171 s	513 s	same pod and card, the off arm launched with `--no-enable-prefix-caching`
later-turn TTFT mean / p50 / p90	11.8 / 6.7 / 22.2 s	32.4 / 24.2 / 81.2 s	same run
first-turn TTFT mean	14.4 s	26.7 s	same run
server prefix-cache hit rate	78%		same run
Needle at 250K: TTFT cold / from the cache	224 to 226 s / 1.1 to 1.7 s		same pod, cold then hit
Needle at 130K / 65K: TTFT cold	69 / 25 to 26 s		same pod
KV pool	623,359 tokens		start-up log

Earlier single measurements on Qwen3.5-27B-FP8 (same geometry and settings), requests that did not hit the cache: single-stream decode 48.8 / 48.0 / 45.0 tokens/s at 2K / 8K / 32K context; cold TTFT 0.53 / 2.10 / 9.96 s at the same lengths; 32 concurrent 12K-token documents with 1,000 output tokens each, all 32 running at once, 32/32 correct, KV peak 80%.

Known issues

Prompts shorter than one 896-token block never hit the cache, and a hit resumes from the last full block before the end of the shared prefix.
Cold outputs with prefix caching on and off are not bit-identical. On the 32 parity prompts, greedy, the two launches agreed on 27/32 outputs; two launches with caching off that differ only in --max-num-batched-tokens (1,792 against 896) agreed on 20/32. That is the noise floor of changing the prefill chunking on this FP8 model, and the caching difference stays inside it.
Temperature 0 still varies with batch composition on HPU; expect two or three trajectory families across repeated runs under load.
Give test requests a real token budget: this model reasons at length, and a short max_tokens returns empty content with finish_reason: length.

What not to do

Do not turn on prefix caching on this release without gdn-prefix-cache: hits then resume the GDN layers from a wrong state.
Do not use lazy mode for this model.
Do not pass {"backend": "xgrammar", "disable_any_whitespace": true} to get past the start-up check: schemas xgrammar cannot compile (multipleOf, uniqueItems, patternProperties, ...) then fail with HTTP 400 instead of falling back to guidance.
Do not raise --max-num-batched-tokens to 3,584 or more at 256K: a cold prefill past about 131K context then died with PT_DEVMEM OOM: No enough memory for defragment (seen with prefix caching on).
Do not set VLLM_QWEN3_STRICT_THINK_END=1 on models that sometimes call tools without closing </think>; it is for models that always close it.