Qwen/Qwen3-235B-A22B-Instruct-2507-FP8

235B MoE (22B active) instruct model, FP8 block-quantized checkpoint, 128K context on four Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide

moe235B / 22B active131,072 ctx servedfp8vllm-gaudi v0.26.0Text

Hardware

4 cards · TP 4

Image

vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1

Variant

The official FP8 checkpoint (236 GB), four Gaudi 2 cards.

Features

Fixes

profile v0.26.0 fp8-moe-gather

13 fixes this configuration depends on; opt-in fixes are named.

Launch

Shell

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3-235B-A22B-Instruct-2507-FP8", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "max_tokens": 512}'

export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_HPU_MOE_GATHER=1
export VLLM_HPU_MOE_GATHER_RATIO=0.25
export VLLM_HPU_MOE_GATHER_VERIFY=0
export VLLM_HPU_FORCE_CHANNEL_FP8=1
export RUNTIME_SCALE_PATCHING=1
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.2
export VLLM_GRAPH_PROMPT_RATIO=0.15
export VLLM_GRAPH_PROMPT_STRATEGY=max_bs
export VLLM_GRAPH_DECODE_STRATEGY=max_bs
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=4
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=8
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=2048
export VLLM_PROMPT_SEQ_BUCKET_MAX=131072
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=8192
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile plus this recipe's opt-in fixes, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 fp8-moe-gather || exit 1

vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.45 \
  --block-size 128 \
  --max-num-batched-tokens 16384 \
  --max-cudagraph-capture-size 2048 \
  --enable-prompt-tokens-details \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --trust-remote-code

Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.

At a glance

Checks that passed

Greedy answers identical to the previous image on the functional set (6/6)
top_logprobs soak 400/400 clean with gather-logprobs-int32 and logprobs-decode-guard
Tool calls, plain and streamed, on the first replicas under live traffic: no 5xx
Thinking twin, same image and recipe: 90-minute soak at 12 in flight (prompts p50 1.2K, p99 35K, shared prefixes), 888 done, 0 errors, 0 restarts, HBM peak 90.1 GB per card

Notes

Without --max-cudagraph-capture-size every prefill shape captures an HPU graph: the 90-minute soak reached 97.5 GB of 98.3 GB per card on v0.26.0 without the cap.
The capture cap costs about 1.2 s on an 8K-token prefill (TTFT 3.75 s vs 2.56 s uncapped); 1K and 32K prompts are unchanged.
Loading is CPU-bound (FP8 rescale per tensor on Gaudi 2); a restart takes about 26 minutes.

Guide

Qwen's 235B mixture-of-experts model (22B active, 128 experts, top 8) in its July 2025 Instruct and Thinking variants, from the official FP8 block-quantized checkpoints, at 128K context on four Gaudi 2 cards.


Models	Qwen/Qwen3-235B-A22B-Instruct-2507-FP8, Qwen/Qwen3-235B-A22B-Thinking-2507-FP8
Hardware	4x Gaudi 2 (TP 4)
Image	vllm-gaudi `v0.26.0` (`b059f9c`), vLLM `568afb3a`, Gaudi SW 1.24.1
Precision	FP8 checkpoint (236 GB)
Context	131,072 served (262,144 native)
Concurrency	`--max-num-seqs 8`
Parsers	tools `hermes`; reasoning `deepseek_r1` (Thinking only)
Recipes	Instruct, Thinking

Differences between the two: Thinking uses --max-num-batched-tokens 32768 (Instruct 16384), a 4,096 prompt-bucket step (Instruct 2,048), the reasoning parser, and two FP8 weight-format variables (VLLM_HPU_CONVERT_TO_FP8UZ=1, PT_HPU_WEIGHT_SHARING=0).

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe).

Fix	Why this model needs it
`prompt-order`	prefix caching can read another request's KV in the same step
`sampling-rows-refresh`	stale device sampling rows after a batch reorder
`structured-output-rows`	structured-output steps sample each row with another request's parameters
`bad-words-index`	`bad_words` raises `IndexError` in the engine core
`logitsprocs-rows`	`logit_bias`, `min_p` and `min_tokens` are silently ignored
`async-discard-counter`	`AttributeError` on the first preemption under async scheduling
`resumed-prefill-logits`	a request resumed after preemption with prefix-cache hits samples again at every recomputed position, so the generated tail repeats
`gather-logprobs-int32`	in lazy mode the `top_logprobs` ids come back corrupted; an out-of-vocabulary id kills the server
`prompt-logprobs`	`qwen3_moe` returns flat hidden states; the first `prompt_logprobs` request kills the engine, and the fix also scores the logits 512 rows at a time instead of a whole 16K to 32K chunk
`logprobs-decode-guard`	a bad logprob id is replaced instead of stopping the API server
`seeded-sampling`	a request with `seed` can segfault a TP worker in lazy mode
`fp8-moe-gather` (opt-in)	gathered-expert FP8 MoE combine (vllm-gaudi PR #1731, merged for v0.30): a decode step reads only the experts it routes to (+30 to 58% at 1 to 4 sequences against the previous image without it)
`rank-exception-fatal`	an exception on a non-zero TP rank is dropped and the engine keeps serving with ranks out of step; the fix stops it

Verified

On vllm-gaudi v0.26.0, four Gaudi 2 cards, the flags above:

Thinking: 90-minute soak at 12 in flight (prompts p50 1.2K, p99 35K tokens, shared 0.5K to 16K prefixes): 888 done, 0 errors, 0 restarts, 113 distinct shapes, HBM peak 90.1 GB per card.
Thinking: reasoning split clean (no <think> in content), non-streamed replies carry the reasoning field, streamed tool calls parse.
Instruct: greedy answers identical to the previous image on the functional set (6/6); top_logprobs soak 400/400 clean; plain and streamed tool calls on live traffic with no 5xx.
Without --max-cudagraph-capture-size, the same 90-minute soak drove HBM to 97.5 GB of 98.3 GB per card, three times above 97 GB.

Performance

Single measurements on four cards, vllm-gaudi v0.26.0 with fp8-moe-gather (gate tokens x top_k <= 32, which is VLLM_HPU_MOE_GATHER_RATIO=0.25 with 128 experts), warm, 256 output tokens per stream, two repetitions.

In flight	Instruct, tokens/s per replica	Thinking, tokens/s per replica
1	22.3 / 22.5	21.9 / 22.0
2	42.5 / 44.2	42.5 / 42.7
4	67.2 / 67.2	67.2 / 67.2
8	104.2 / 106.9	102.7 / 105.8

Inter-token latency on live traffic: p50 63 ms, p95 94 ms with 2 to 5 sequences running. TTFT for an 8K-token prompt is 3.75 s with the capture cap and 2.56 s without; 1K and 32K prompts are unchanged.

Known issues

Loading is CPU-bound: on Gaudi 2 every FP8 tensor is rescaled on the host, so a restart takes about 26 minutes even from a fast weight store. vllm-gaudi sizes its load thread pool from the host's core count; under a container CPU quota that pool is throttled. Run without a CPU limit, or cap the pool at the quota.
The Thinking soak left 8.2 GB of HBM headroom per card. If it ever runs out, lower --max-num-batched-tokens to 16384 first.
Penalties are not applied under async scheduling on v0.26.0 (see Qwen3-30B-A3B-2507).

What not to do

Do not drop --max-cudagraph-capture-size. HBM then grows with every new prefill shape until PT_DEVMEM allocation failures in graph replay restart the engine.
Do not raise VLLM_HPU_MOE_GATHER_RATIO to 0.5 (a gate of 64, the value used for 256-expert models). With 128 experts and 5 or more sequences the gather copies most of the expert set and loses to the stock op; 0.25 (a gate of 32) keeps the gain at 1 to 4 sequences and ties the stock op at 8.
Do not lower Thinking's --max-num-batched-tokens to 16384 for speed: on the A/B it was equal on normal traffic and slower at long context (4 x 20K-token prompts, TTFT 6.4 s vs 3.5 to 5.5 s).
Do not measure from cold recipe caches: a cold first pass showed a fake 14x difference between two chunk sizes that disappeared on the warm repeat. Warm every prompt-length bucket you compare.