Gaudi Cookbook / Recipes
/

Qwen/Qwen-AgentWorld-35B-A3B

35B hybrid MoE (Gated DeltaNet + attention, 3B active) agent model, BF16, native 256K context on two Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide
moe35B / 3B active262,144 ctx servedbf16vllm-gaudi v0.26.0Text
Hardware
2 cards · TP 2
Image
vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1
Variant
BF16 weights (69 GB), two Gaudi 2 cards.
Features
Fixes
profile v0.26.0
11 fixes this configuration depends on; opt-in fixes are named.
Launch
Shell 
export PT_HPU_LAZY_MODE=0
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_USE_NAIVE_MAMBA_CACHE_SHARING=1
export VLLM_COMPACT_GDN=1
export VLLM_SKIP_WARMUP=true
export VLLM_ENGINE_ITERATION_TIMEOUT_S=7200
export VLLM_RPC_TIMEOUT=7200000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 || exit 1

vllm serve Qwen/Qwen-AgentWorld-35B-A3B \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.9 \
  --max-num-batched-tokens 1024 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.
At a glance

Checks that passed

  • Known-answer prompts 8/8 at temperature 0 and 16/16 with 16 in flight
  • Tool calls: 5/5 valid calls sampled at 0.6, streamed call assembled, tool result used in the answer
  • top_logprobs 20, prompt_logprobs, JSON schema and logit_bias all take effect; server healthy after each
  • Needles, 3 depths each, cold: 3/3 at 65K, 131K and 244K tokens
  • Single stream 26.3 tokens/s, 16 streams 321.4 tokens/s aggregate (256 tokens, end to end)

Notes

  • Prefix caching is off, as verified. On the stock release prefix-cache hits on Gated DeltaNet layers resume from the wrong state; gdn-prefix-cache makes them correct on other Qwen3.5-family recipes but has not been run on this model.
  • On easy arithmetic the model often answers itself instead of calling an offered calculator tool; this is model behaviour, not parsing.
Guide

Qwen's 35B hybrid MoE agent model (30 Gated DeltaNet and 10 attention layers, 256 experts, 3B active) in BF16, text only, at its native 256K context on two Gaudi 2 cards, 16 sequences at a time. Prefix caching is off on this release (see Prefix caching).
Model Qwen/Qwen-AgentWorld-35B-A3B
Hardware 2x Gaudi 2 (TP 2)
Image vllm-gaudi v0.26.0 (b059f9c), vLLM 568afb3a, Gaudi SW 1.24.1
Precision BF16 (69 GB of weights)
Context 262,144 (native)
Concurrency --max-num-seqs 16
Parsers tools qwen3_coder, reasoning qwen3
Recipe recipes/Qwen/Qwen-AgentWorld-35B-A3B/recipe.yaml

Prefix caching

Prefix caching is off: on this release, prefix-cache hits on Gated DeltaNet layers resume from the wrong state. Upstream vllm-gaudi main has merged prefix caching for the non-compact layout; use a release that includes it before turning prefix caching on.

Required fixes

All from Fixes, all in --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe). The profile also runs seeded-sampling, prompt-order and resumed-prefill-logits; this recipe does not depend on them (the seed segfault is a lazy-mode bug, and prefix caching is off).

Fix Why this model needs it
sampling-rows-refresh stale device sampling rows after a batch reorder
structured-output-rows structured-output steps sample each row with another request's parameters
bad-words-index bad_words raises IndexError in the engine core
logitsprocs-rows logit_bias, min_p and min_tokens are silently ignored
async-discard-counter AttributeError on the first preemption under async scheduling
hybrid-decode-pad when the hybrid KV block is split into 128-token kernel blocks, decode padding writes into a live block of another request
gather-logprobs-int32 the id corruption is lazy-mode only, but the fix is harmless here and keeps one fix set across recipes
prompt-logprobs qwen3_5_moe returns flat hidden states; the first prompt_logprobs request kills the engine
logprobs-decode-guard a bad logprob id from any source is replaced instead of stopping the API server
rank-exception-fatal an exception on a non-zero TP rank stops the engine instead of serving from out-of-step ranks
rope-cpu-cache Gaudi fp32 cos/sin are wrong past about 1.5e5, so RoPE tables built on the HPU are wrong from position about 196K

Verified

On vllm-gaudi v0.26.0, two Gaudi 2 cards, with exactly the launch command above in a container started from the stock image (the fixes applied at start-up). Requests went to the OpenAI-compatible API.

Check Result
Known-answer prompts (arithmetic, facts, translation), temperature 0 8/8
The same prompts, 16 in flight 16/16
Tool call, tool_choice: auto, sampled at 0.6 5/5 valid get_weather calls with the right argument
Streamed tool call name and JSON arguments assembled correctly
Tool result fed back the answer uses the returned values
top_logprobs: 20 20 entries on every token
prompt_logprobs: 5 returned; server healthy afterwards
response_format JSON schema valid object
logit_bias +100 on one token takes effect
Needle in a haystack, 3 depths, cold 3/3 at 65K, 131K and 244K tokens
Server health after all checks healthy, no restarts

KV pool at start-up: 5,197,487 tokens.

Performance

Single runs on the server above (measured as: per-call, one server; times are end to end and include time to first token).

Test Result
One stream, short prompt, 256 tokens 26.3 tokens/s
16 streams, short prompts, 256 tokens each 321.4 tokens/s aggregate, 20.1 per stream
Cold prompt of 65K tokens (needle) 17 s (99 s for the first, which also compiled its graphs) to the answer
Cold prompt of 131K tokens (needle) 33 to 46 s to the answer
Cold prompt of 244K tokens (needle) 72 to 85 s to the answer

Known issues

  • Offered a calculator tool, the model often works out easy arithmetic itself instead of calling it. The tools are rendered and parsed correctly; it calls the tool when asked to, or for harder sums. This is model behaviour.
  • Give test requests a real token budget: the model reasons before answering.

What not to do

  • Do not drop --language-model-only.
  • Do not use lazy mode.
  • Do not turn on prefix caching on this release.