Qwen/Qwen3.5-122B-A10B-FP8

122B hybrid MoE (Gated DeltaNet + attention, 10B active) reasoning model, FP8, native 256K context on four Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide

moe122B / 10B active262,144 ctx servedfp8vllm-gaudi v0.26.0Text

Hardware

4 cards · TP 4 · EP

Image

vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1

Variant

The official FP8 checkpoint (block 128 x 128, dynamic activations, 127 GB), four Gaudi 2 cards.

Features

Fixes

profile v0.26.0 gdn-prefix-cache fp8-moe-gather

15 fixes this configuration depends on; opt-in fixes are named.

Launch

Shell

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3.5-122B-A10B-FP8", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "max_tokens": 512}'

export PT_HPU_LAZY_MODE=0
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_HPU_MOE_GATHER=1
export VLLM_HPU_MOE_GATHER_RATIO=1.0
export VLLM_HPU_MOE_GATHER_VERIFY=0
export VLLM_USE_NAIVE_MAMBA_CACHE_SHARING=1
export VLLM_COMPACT_GDN=1
export PT_HPUGRAPH_DISABLE_TENSOR_CACHE=false
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.3
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=16
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=16
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=4096
export VLLM_PROMPT_SEQ_BUCKET_MAX=262144
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=16384
export VLLM_ENGINE_ITERATION_TIMEOUT_S=7200
export VLLM_RPC_TIMEOUT=7200000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile plus this recipe's opt-in fixes, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 gdn-prefix-cache fp8-moe-gather || exit 1

vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.75 \
  --block-size 128 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --language-model-only

Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.

At a glance

Checks that passed

Known-answer prompts 8/8 at temperature 0 and 16/16 with 16 in flight
Tool calls: 5/5 valid calls sampled at 0.6, streamed call assembled, tool result used in the answer
top_logprobs 20, prompt_logprobs, JSON schema and logit_bias all take effect; server healthy after each
Prefix-cache hit vs cold, 8 greedy prompts of 6K to 126K tokens: identical text and bit-identical logprobs 8/8; cached tokens a multiple of 1,152 on every hit
Needles, 3 depths each, cold: 3/3 at 65K, 131K and 244K tokens
Single stream 43.8 tokens/s, 16 streams 315.6 tokens/s aggregate (256 tokens, end to end)

Notes

Prefix caching needs gdn-prefix-cache: on the stock release, prefix-cache hits on Gated DeltaNet layers resume from the wrong state. The fix stores GDN state checkpoints in the KV pages (1152-token blocks here) and keeps the compact GDN slots (VLLM_COMPACT_GDN=1).
VLLM_HPU_MOE_GATHER_RATIO is per card: with expert parallelism each card holds 64 experts, so 1.0 keeps the gather path up to 8 decode tokens. Above that the stock MoE op runs.
Loading under a CPU quota is slow: vllm-gaudi sizes its load thread pool from the host's core count, and the per-tensor FP8 rescale is then CPU-throttled. Run without a CPU limit, or cap the pool at the quota.

Guide

Qwen's 122B hybrid MoE (Gated DeltaNet and attention layers, 256 experts, 10B active) reasoning model from the official FP8 checkpoint, text only, at its native 256K context on four Gaudi 2 cards with expert parallelism. Prefix caching is on through the gdn-prefix-cache fix.


Model	Qwen/Qwen3.5-122B-A10B-FP8
Hardware	4x Gaudi 2 (TP 4, expert parallel)
Image	vllm-gaudi `v0.26.0` (`b059f9c`), vLLM `568afb3a`, Gaudi SW 1.24.1
Precision	FP8 checkpoint (block 128 x 128, dynamic activations, 127 GB)
Context	262,144 (native)
Concurrency	`--max-num-seqs 16`
Parsers	tools `qwen3_coder`, reasoning `qwen3`
Recipe	`recipes/Qwen/Qwen3.5-122B-A10B-FP8/recipe.yaml`

Prefix caching

On the stock release, prefix-cache hits on the Gated DeltaNet layers resume from the wrong state, so prefix caching is only correct with gdn-prefix-cache. The fix keeps the GDN layers in their compact per-request slots (VLLM_COMPACT_GDN=1) and stores a state checkpoint at the end of every full KV block in that block's own attention page. A hit restores the checkpoint before the next chunk, and the full KV pool stays available. With caching on, the hybrid KV block is 1,152 tokens and prefill chunks end on block boundaries; prompts shorter than one block never hit. The design and the upstream status are in Prefix caching on Gated DeltaNet hybrids.

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe). The profile also runs seeded-sampling; this recipe does not depend on it (the seed segfault is a lazy-mode bug).

Fix	Why this model needs it
`prompt-order`	with prefix caching, a request that hits a block another request writes in the same step read it before it was written
`sampling-rows-refresh`	stale device sampling rows after a batch reorder
`structured-output-rows`	structured-output steps sample each row with another request's parameters
`bad-words-index`	`bad_words` raises `IndexError` in the engine core
`logitsprocs-rows`	`logit_bias`, `min_p` and `min_tokens` are silently ignored
`async-discard-counter`	`AttributeError` on the first preemption under async scheduling
`resumed-prefill-logits`	a request resumed after preemption with prefix-cache hits samples again at every recomputed position, so the generated tail repeats
`hybrid-decode-pad`	when the hybrid KV block is split into 128-token kernel blocks, decode padding writes into a live block of another request
`gdn-prefix-cache` (opt-in)	prefix-cache hits on the GDN layers resume from a wrong state on the stock release; the fix stores and restores GDN checkpoints in the KV pages and zeroes new blocks and slots. It needs `hybrid-decode-pad` first
`gather-logprobs-int32`	the id corruption is lazy-mode only, but the fix is harmless here and keeps one fix set across recipes
`prompt-logprobs`	`qwen3_5_moe` returns flat hidden states (also with `--language-model-only`); the first `prompt_logprobs` request kills the engine
`logprobs-decode-guard`	a bad logprob id from any source is replaced instead of stopping the API server
`fp8-moe-gather` (opt-in)	gathered-expert FP8 MoE, a backport from v0.30.0: faster low-batch decode
`rope-cpu-cache`	Gaudi fp32 cos/sin are wrong past about 1.5e5, so RoPE tables built on the HPU are wrong from position about 196K
`rank-exception-fatal`	an exception on a non-zero TP rank stops the engine instead of serving from out-of-step ranks

Verified

On vllm-gaudi v0.26.0, four Gaudi 2 cards, with exactly the launch command above in a container started from the stock image (the fixes applied at start-up). Requests went to the OpenAI-compatible API.

Check	Result
Known-answer prompts (arithmetic, facts, translation), temperature 0	8/8
The same prompts, 16 in flight	16/16
Tool call, `tool_choice: auto`, sampled at 0.6	5/5 valid `get_weather` calls with the right argument
Streamed tool call	name and JSON arguments assembled correctly
Tool result fed back	the answer uses the returned values
`top_logprobs: 20`	20 entries on every token
`prompt_logprobs: 5`	returned; server healthy afterwards
`response_format` JSON schema	valid object
`logit_bias` +100 on one token	takes effect
Prefix-cache hit vs cold (`cache_salt`), greedy, 8 prompts of 6K to 126K tokens with a needle	identical text and bit-identical logprobs 8/8; cached tokens a multiple of 1,152 on every hit
Needle in a haystack, 3 depths, cold	3/3 at 65K, 131K and 244K tokens
Server health after all checks	healthy, no restarts

KV pool at start-up: 2,885,824 tokens.

Performance

Single runs on the server above (measured as: per-call, one server; times are end to end and include time to first token).

Test	Result
One stream, short prompt, 256 tokens	43.8 tokens/s
16 streams, short prompts, 256 tokens each	315.6 tokens/s aggregate, 19.7 per stream
Cold prompt of 65K tokens (needle)	16 to 25 s to the answer
Cold prompt of 131K tokens (needle)	24 to 33 s to the answer
Cold prompt of 244K tokens (needle)	56 to 82 s to the answer

Known issues

Loading is slow under a CPU quota: vllm-gaudi sizes its load thread pool from the host's core count, and the per-tensor FP8 rescale on Gaudi 2 is then CPU-throttled. Run without a CPU limit, or cap the pool at the quota.
Give test requests a real token budget: this model reasons at length.

What not to do

Do not use lazy mode for this model.
Do not turn on prefix caching without gdn-prefix-cache, and do not set VLLM_COMPACT_GDN=0 with it.
Do not copy VLLM_HPU_MOE_GATHER_RATIO to a layout without expert parallelism: the ratio is relative to the experts on one card.