Qwen/Qwen3-30B-A3B-Thinking-2507

30B MoE (3B active) reasoning model, always thinks, served at 128K context on one Gaudi 2 card in BF16.

Verified on Gaudi 2 Hugging Face Full guide

moe30.5B / 3.3B active131,072 ctx servedbf16vllm-gaudi v0.26.0Text

Hardware

1 card · TP 1

Image

vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1

Variant

BF16 weights (61 GB), one Gaudi 2 card.

Features

Fixes

profile v0.26.0

11 fixes this configuration depends on; opt-in fixes are named.

Launch

Shell

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3-30B-A3B-Thinking-2507", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "max_tokens": 512}'

export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.4
export VLLM_GRAPH_PROMPT_RATIO=0.5
export VLLM_GRAPH_PROMPT_STRATEGY=max_bs
export VLLM_GRAPH_DECODE_STRATEGY=max_bs
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=8
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=8
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=4096
export VLLM_PROMPT_SEQ_BUCKET_MAX=131072
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=32768
export OMP_NUM_THREADS=1
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 || exit 1

vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507 \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.9 \
  --block-size 128 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prompt-tokens-details \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --trust-remote-code

Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.

At a glance

Checks that passed

Production-shaped load (about 19K-token prompts, natural reasoning output): 12/12 at 2, 16/16 at 4, 24/24 at 8 in flight, 0 failures
Long-context arm (1 in 8 prompts at about 105K tokens) at 8 in flight: 32/32 with 5 preemptions, 0 restarts
Without async-discard-counter the same arm killed the engine on the first preemption

Notes

Give test requests a real max_tokens: a short budget is spent inside the reasoning and content comes back empty.
KV pool is 221,312 tokens at these settings (1.69 x 128K).
Keep --max-num-batched-tokens at 2048: on this architecture a padded prefill tile near 2^29 elements (8192-token chunk at 32K+ context) returns garbage or kills the engine.

Guide

Qwen's 30B mixture-of-experts model (3.3B active), in its July 2025 non-thinking (Instruct) and always-thinking (Thinking) variants, served in BF16 at 128K context on a single Gaudi 2 card.


Models	Qwen/Qwen3-30B-A3B-Instruct-2507, Qwen/Qwen3-30B-A3B-Thinking-2507
Hardware	1x Gaudi 2 (TP 1)
Image	vllm-gaudi `v0.26.0` (`b059f9c`), vLLM `568afb3a`, Gaudi SW 1.24.1
Precision	BF16 (61 GB of weights)
Context	131,072 served (262,144 native)
Concurrency	`--max-num-seqs 8`
Parsers	tools `hermes`; reasoning `deepseek_r1` (Thinking only)
Recipes	Instruct, Thinking

The two variants are configuration twins. Thinking adds --reasoning-parser deepseek_r1: its chat template opens <think> in the prompt, so the reply starts inside the reasoning without an opening tag, which deepseek_r1 handles.

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe).

Fix	Why this model needs it
`prompt-order`	prefix caching (on by default) can read another request's KV in the same step
`sampling-rows-refresh`	after a batch reorder the device sampling rows are stale (wrong temperature per row)
`structured-output-rows`	structured-output steps sample each row with another request's parameters
`bad-words-index`	`bad_words` indexes the wrong rows and raises `IndexError` in the engine core
`logitsprocs-rows`	`logit_bias`, `min_p` and `min_tokens` are silently ignored without it
`async-discard-counter`	v0.26.0 renamed the counter the HPU async scheduler reads: `AttributeError` on the first preemption
`resumed-prefill-logits`	a request resumed after preemption with prefix-cache hits samples again at every recomputed position, so the generated tail repeats
`gather-logprobs-int32`	in lazy mode the `top_logprobs` ids come back corrupted; an out-of-vocabulary id kills the server
`prompt-logprobs`	`qwen3_moe` returns flat hidden states; the first `prompt_logprobs` request kills the engine
`logprobs-decode-guard`	a bad logprob id is replaced instead of stopping the API server
`seeded-sampling`	a request with `seed` uses an HPU generator that can segfault the worker in lazy mode

Verified

On vllm-gaudi v0.26.0, one Gaudi 2 card, the flags above:

Thinking, production-shaped load (about 19K-token prompts, natural reasoning output): 12/12 at 2, 16/16 at 4 and 24/24 at 8 requests in flight, 0 failures.
Thinking, 8 in flight with 1 in 8 prompts at about 105K tokens: 32/32 with 5 preemptions and 0 restarts. Without async-discard-counter the first preemption killed the engine.
Instruct, against the previous image with the same flags: tool calls, an 84K-token needle and a 400-request top_logprobs soak (0 corrupt) all pass.
Instruct, prompt_logprobs: 68/68 prompt sizes times prompt_logprobs values, greedy parity 10/10 bit-identical with the fix.
Instruct, prefix-cache isolation with prompt-order: 48/48, 0 errors.

Performance

Single measurements on one card, vllm-gaudi v0.26.0, the Thinking variant (same architecture and flags as Instruct).

In flight	Prompt / output	Tokens/s per stream	Tokens/s per card
1	400 / 1,500	82	82
2	400 / 1,500	66	126
4	400 / 1,500	50	149
8	400 / 1,500	38	261
2	about 19K / natural reasoning	49.3	69.5
4	about 19K / natural reasoning	34.0	93.9
8	about 19K / natural reasoning	22.7	143.0

The long-prompt rows are lower per stream because each new 19K prompt is prefilled in 2,048-token chunks between decode steps. The KV pool holds 221,312 tokens (1.69 full 128K requests).

Known issues

Penalties (repetition_penalty, presence_penalty, frequency_penalty) are not applied under async scheduling on v0.26.0: the HPU runner never appends sampled ids to the output history the penalties read. Use --no-async-scheduling together with a penalty-shape fix if a client depends on them.
The first long prompt on a fresh node compiles its buckets (132 s for an 84K-token prompt, 5.6 s once warm). The timeouts in the environment are sized for that.

What not to do

Do not raise --max-num-batched-tokens above 2,048. On this architecture a prefill tile padded to 8,192 tokens at 32K+ context (near 2^29 elements) returns garbage (greedy) or kills the engine (sampling); see Qwen3-Coder-30B-A3B.
Do not drop async-discard-counter and resumed-prefill-logits because a low-concurrency test passed: at 2 sequences this model never preempted, at 8 it does, and each preemption without them either kills the engine or corrupts the resumed stream.
Do not judge the Thinking variant with a small max_tokens: the budget is spent inside the reasoning and content comes back empty, which looks like a serving bug and is not.