Qwen3-30B-A3B-2507 (Instruct and Thinking)¶

Qwen's 30B mixture-of-experts model (3.3B active), in its July 2025 non-thinking (Instruct) and always-thinking (Thinking) variants, served in BF16 at 128K context on a single Gaudi 2 card.


Models	Qwen/Qwen3-30B-A3B-Instruct-2507, Qwen/Qwen3-30B-A3B-Thinking-2507
Hardware	1x Gaudi 2 (TP 1)
Image	vllm-gaudi `v0.26.0` (`b059f9c`), vLLM `568afb3a`, Gaudi SW 1.24.1
Precision	BF16 (61 GB of weights)
Context	131,072 served (262,144 native)
Concurrency	`--max-num-seqs 8`
Parsers	tools `hermes`; reasoning `deepseek_r1` (Thinking only)
Recipes	Instruct, Thinking

The two variants are configuration twins. Thinking adds --reasoning-parser deepseek_r1: its chat template opens <think> in the prompt, so the reply starts inside the reasoning without an opening tag, which deepseek_r1 handles.

Launch¶

$COOKBOOK is a checkout of this repository inside the container.

InstructThinking

export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.4
export VLLM_GRAPH_PROMPT_RATIO=0.5
export VLLM_GRAPH_PROMPT_STRATEGY=max_bs
export VLLM_GRAPH_DECODE_STRATEGY=max_bs
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=8
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=8
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=4096
export VLLM_PROMPT_SEQ_BUCKET_MAX=131072
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=32768
export OMP_NUM_THREADS=1
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 || exit 1

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.9 \
  --block-size 128 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --trust-remote-code

export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.4
export VLLM_GRAPH_PROMPT_RATIO=0.5
export VLLM_GRAPH_PROMPT_STRATEGY=max_bs
export VLLM_GRAPH_DECODE_STRATEGY=max_bs
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=8
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=8
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=4096
export VLLM_PROMPT_SEQ_BUCKET_MAX=131072
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=32768
export OMP_NUM_THREADS=1
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 || exit 1

vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507 \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.9 \
  --block-size 128 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prompt-tokens-details \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --trust-remote-code

Buckets¶

No bucket file. The ranges are in the environment: batch buckets 1 to 8 (matching --max-num-seqs 8), prompt length 128 to 131,072 in steps of 4,096 with exponential bucketing, decode blocks 128 to 32,768 in steps of 1,024.

Required fixes¶

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe).

Fix	Why this model needs it
`prompt-order`	prefix caching (on by default) can read another request's KV in the same step
`sampling-rows-refresh`	after a batch reorder the device sampling rows are stale (wrong temperature per row)
`structured-output-rows`	structured-output steps sample each row with another request's parameters
`bad-words-index`	`bad_words` indexes the wrong rows and raises `IndexError` in the engine core
`logitsprocs-rows`	`logit_bias`, `min_p` and `min_tokens` are silently ignored without it
`async-discard-counter`	v0.26.0 renamed the counter the HPU async scheduler reads: `AttributeError` on the first preemption
`resumed-prefill-logits`	a request resumed after preemption with prefix-cache hits samples again at every recomputed position, so the generated tail repeats
`gather-logprobs-int32`	in lazy mode the `top_logprobs` ids come back corrupted; an out-of-vocabulary id kills the server
`prompt-logprobs`	`qwen3_moe` returns flat hidden states; the first `prompt_logprobs` request kills the engine
`logprobs-decode-guard`	a bad logprob id is replaced instead of stopping the API server
`seeded-sampling`	a request with `seed` uses an HPU generator that can segfault the worker in lazy mode

Verified¶

On vllm-gaudi v0.26.0, one Gaudi 2 card, the flags above:

Thinking, production-shaped load (about 19K-token prompts, natural reasoning output): 12/12 at 2, 16/16 at 4 and 24/24 at 8 requests in flight, 0 failures.
Thinking, 8 in flight with 1 in 8 prompts at about 105K tokens: 32/32 with 5 preemptions and 0 restarts. Without async-discard-counter the first preemption killed the engine.
Instruct, against the previous image with the same flags: tool calls, an 84K-token needle and a 400-request top_logprobs soak (0 corrupt) all pass.
Instruct, prompt_logprobs: 68/68 prompt sizes times prompt_logprobs values, greedy parity 10/10 bit-identical with the fix.
Instruct, prefix-cache isolation with prompt-order: 48/48, 0 errors.

Performance¶

Single measurements on one card, vllm-gaudi v0.26.0, the Thinking variant (same architecture and flags as Instruct).

In flight	Prompt / output	Tokens/s per stream	Tokens/s per card
1	400 / 1,500	82	82
2	400 / 1,500	66	126
4	400 / 1,500	50	149
8	400 / 1,500	38	261
2	about 19K / natural reasoning	49.3	69.5
4	about 19K / natural reasoning	34.0	93.9
8	about 19K / natural reasoning	22.7	143.0

The long-prompt rows are lower per stream because each new 19K prompt is prefilled in 2,048-token chunks between decode steps. The KV pool holds 221,312 tokens (1.69 full 128K requests).

Known issues¶

Penalties (repetition_penalty, presence_penalty, frequency_penalty) are not applied under async scheduling on v0.26.0: the HPU runner never appends sampled ids to the output history the penalties read. Use --no-async-scheduling together with a penalty-shape fix if a client depends on them.
The first long prompt on a fresh node compiles its buckets (132 s for an 84K-token prompt, 5.6 s once warm). The timeouts in the environment are sized for that.

What not to do¶

Do not raise --max-num-batched-tokens above 2,048. On this architecture a prefill tile padded to 8,192 tokens at 32K+ context (near 2^29 elements) returns garbage (greedy) or kills the engine (sampling); see Qwen3-Coder-30B-A3B.
Do not drop async-discard-counter and resumed-prefill-logits because a low-concurrency test passed: at 2 sequences this model never preempted, at 8 it does, and each preemption without them either kills the engine or corrupts the resumed stream.
Do not judge the Thinking variant with a small max_tokens: the budget is spent inside the reasoning and content comes back empty, which looks like a serving bug and is not.