Gaudi Cookbook / Recipes
/

meta-llama/Llama-4-Scout-17B-16E-Instruct

109B MoE (17B active, 16 experts, top-1) multimodal instruct model, BF16, 128K context on four Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide
moe109B / 17B active131,072 ctx servedbf16vllm-gaudi v0.26.0Multimodal
Hardware
4 cards · TP 4
Image
vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1
Variant
BF16 weights (217 GB), four Gaudi 2 cards.
Features
Fixes
profile v0.26.0moe-router-weight-on-inputllama4-tools-lenient
14 fixes this configuration depends on; opt-in fixes are named.
Launch
Shell 
export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export PT_HPUGRAPH_DISABLE_TENSOR_CACHE=1
export VLLM_SKIP_WARMUP=true
export VLLM_MULTIMODAL_BUCKETS=None
export VLLM_GRAPH_RESERVED_MEM=0.4
export VLLM_GRAPH_PROMPT_RATIO=0.5
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_HPU_MOE_WEIGHT_ON_INPUT=1
export LLAMA4_TOOL_CALL_POLICY=independent
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000

# Fixes: the v0.26.0 profile plus this recipe's opt-in fixes, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 \
    moe-router-weight-on-input llama4-tools-lenient || exit 1

vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.9 \
  --max-num-batched-tokens 2048 \
  --max-cudagraph-capture-size 2048 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice \
  --chat-template "$COOKBOOK/fixes/llama4-tools-lenient/llama4-tools.jinja" \
  --default-chat-template-kwargs '{"fmt": "tag", "sys_mode": "meta", "defs": "meta"}' \
  --tool-parser-plugin "$COOKBOOK/fixes/llama4-tools-lenient/llama4_tools_lenient.py" \
  --tool-call-parser llama4_lenient \
  --limit-mm-per-prompt '{"image":4,"video":0}'
Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.
At a glance

Checks that passed

  • Next-token KL against a float32 CPU reference (59 positions, prompts of 37, 7.6K and 9.7K tokens): 0.003 / 0.006 / 0.007 with the router fix, 0.029 / 0.354 / 0.098 without
  • Coding-agent tool scenarios straight at the server: 23/24 with the tools template, parser and router fix; 0/16 with the stock template and parser and no router fix
  • Tool reliability 120/120 sampled at 0.6; with seeded requests 120/120
  • Prefix-cache hits vs cold: 29/32, equal to the noise floor of shifting the chunk boundary by one token (29/32 with no cache)
  • Image prefix-cache gate (same image new text, new image same text, swapped order, 4 images, bursts): 114 checks, 0 wrong reuse, 0 wrong answers
  • Needles at 43K, 65K and 129.5K tokens: 9/9 cold, 9/9 from cache
  • Greedy top-1 agreement with the previous image 0.9964 over 11,188 tokens; 24/24 greedy texts identical to 512 tokens; vision 6/6
  • 15-minute seeded storm (48 clients, 75% seeded, streaming, aborts): 1,991 requests, 0 errors, 0 restarts

Notes

  • Fix paths in serve_args are relative to a checkout of this repository; point them at wherever the fixes pack lives in the container.
  • With prefix caching on, vllm-gaudi turns off contiguous PA: single-stream decode is 6-11% slower than with caching off; 4+ streams are equal or faster.
  • Penalties (repetition, presence, frequency) are not applied under async scheduling on v0.26.0.
Guide

Meta's 109B mixture-of-experts multimodal model (17B active, 16 experts, top 1), served in BF16 at 128K context on four Gaudi 2 cards with prefix caching (images included) and a tool format the model follows reliably.
Model meta-llama/Llama-4-Scout-17B-16E-Instruct (gated)
Hardware 4x Gaudi 2 (TP 4)
Image vllm-gaudi v0.26.0 (b059f9c), vLLM 568afb3a, Gaudi SW 1.24.1
Precision BF16 (217 GB of weights)
Context 131,072 served (the checkpoint allows 10M)
Concurrency --max-num-seqs 16, up to 4 images per prompt
Parsers tools llama4_lenient (plugin from llama4-tools-lenient)
Recipe recipes/meta-llama/Llama-4-Scout-17B-16E-Instruct/recipe.yaml

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe).

Fix Why this model needs it
prompt-order prefix caching can read another request's KV in the same step
sampling-rows-refresh stale device sampling rows after a batch reorder
structured-output-rows structured-output steps sample each row with another request's parameters
bad-words-index bad_words raises IndexError in the engine core
logitsprocs-rows logit_bias, min_p and min_tokens are silently ignored
async-discard-counter AttributeError on the first preemption under async scheduling
resumed-prefill-logits a request resumed after preemption with prefix-cache hits samples again at every recomputed position, so the generated tail repeats
gather-logprobs-int32 in lazy mode the top_logprobs ids come back corrupted; an out-of-range id reaching the detokenizer killed a replica
prompt-logprobs the first prompt_logprobs request kills the engine
logprobs-decode-guard a bad logprob id is replaced instead of stopping the API server
seeded-sampling a request with seed can segfault a TP worker in lazy mode
moe-router-weight-on-input (opt-in) Llama 4 applies the top-1 router weight to the expert input (expert(w * x)); the HPU fused MoE applied it to the output (w * expert(x)), which differs through SwiGLU on every layer. Enabled by VLLM_HPU_MOE_WEIGHT_ON_INPUT=1
rank-exception-fatal an exception on a non-zero TP rank stops the engine instead of leaving ranks out of step
llama4-tools-lenient (opt-in) the stock template and llama4_pythonic parser fail coding-agent tasks (prose answers, invalid Python calls once a value holds a quote or newline)

Verified

On vllm-gaudi v0.26.0, four Gaudi 2 cards, the flags above unless noted:

  • Router fix against a float32 CPU reference built from the Hugging Face modules (next-token top-20, 59 positions, prompts of 37, 7.6K and 9.7K tokens), KL divergence 0.003 / 0.006 / 0.007; 0.029 / 0.354 / 0.098 without the fix (earlier image). Top-1 agreement after the fix 13/14, 14/14, 31/31.
  • Coding-agent tool scenarios straight at the server: 23/24 with the template, parser and router fix; 0/16 with the stock template and parser and no router fix (earlier image).
  • Tool reliability 120/120 sampled at 0.6, also with seeded requests; tools 7/7.
  • Prefix-cache hits against cold runs: 29/32, the same as shifting the chunk boundary by one token with no cache (top-1 routing flips experts at near-ties). Same-step read-before-write 3 to 4 of 24 against 4 of 24 for a no-sharing control; 0 leaks in 24 isolation rounds.
  • Image prefix-cache gate (same image with new text, new image with the same text, swapped order, 4 images, bursts): 114 checks, 0 wrong reuse, 0 wrong answers.
  • Needles at 43K, 65K and 129.5K tokens: 9/9 cold and 9/9 from cache. KV driven to 100%: 9 checks after preemption, 0 argmax mismatches.
  • Against the previous image: greedy top-1 agreement 0.9964 over 11,188 tokens, 24/24 greedy texts identical to 512 tokens, vision 6/6.
  • 15-minute storm at saturation (48 clients, 75% seeded, streaming, client aborts): 1,991 requests, 0 errors, 0 restarts.

Performance

Single measurements on four cards, vllm-gaudi v0.26.0, 8K-token prompts and 400 output tokens, warm second pass.

Test Result
1 stream 56.2 tokens/s
4 / 8 / 16 streams, per replica 237.3 / 402.2 / 669 tokens/s
TTFT at 8K, cold 0.61 s
TTFT at 32K, cold / repeated (cache hit) 4.38 s / 0.23 s
TTFT at 129.5K, cache hit 0.4 s (about 54 s cold on the earlier image)

The KV pool holds 430,592 tokens (3,364 blocks of 128).

Known issues

  • With prefix caching on, vllm-gaudi turns off contiguous paged attention: single-stream decode is 6 to 11% slower than with caching off (17.8 vs 16.7 ms per token). From 4 streams up it is equal or faster.
  • Remaining tool failures are model slips: a dropped quote, a typo inside JSON arguments, an edit with an empty oldString.
  • Penalties are not applied under async scheduling on v0.26.0 (see Qwen3-30B-A3B-2507).

What not to do

  • Do not set PT_HPUGRAPH_DISABLE_TENSOR_CACHE to anything but "1" on this model (see above). The same setting is not safe on every vision model; on Scout, image requests after a 130K-token prefill were checked and stay correct.
  • Do not use llama4_json-style tool calls: Scout copies the schema into its arguments and wraps calls in code fences.
  • Do not skip moe-router-weight-on-input because tool pass rates look acceptable without it: it is a numerics fix for every reply, not only tool calls.