Gaudi Cookbook / Recipes
/

poolside/Laguna-S-2.1

118B MoE (8B active) coding and agent model with reasoning, BF16, 128K context on four Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide
moe118B / 8B active131,072 ctx servedbf16vllm-gaudi v0.30.0Text
Hardware
4 cards · TP 4
Image
vllm-gaudi releases/v0.30.0 @ 058f107 · vLLM @ ced6857a · Gaudi SW 1.24.1
Variant
BF16 weights (235 GB), four Gaudi 2 cards.
Features
Fixes
profile v0.30.0
12 fixes this configuration depends on; opt-in fixes are named.
Launch
Shell 
export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_SKIP_WARMUP=true
export VLLM_EXPONENTIAL_BUCKETING=False
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=4
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=1024
export VLLM_PROMPT_SEQ_BUCKET_MAX=2048
export VLLM_PROMPT_CTX_BUCKET_MIN=0
export VLLM_PROMPT_CTX_BUCKET_STEP=16
export VLLM_PROMPT_CTX_BUCKET_MAX=1008
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=4
export VLLM_DECODE_BS_BUCKET_MAX=16
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=5120
export VLLM_GRAPH_RESERVED_MEM=0.2
export OMP_NUM_THREADS=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_ENGINE_ITERATION_TIMEOUT_S=7200
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600
export VLLM_RPC_TIMEOUT=7200000

# Fixes: the v0.30.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.30.0 || exit 1

vllm serve poolside/Laguna-S-2.1 \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-seqs 16 \
  --block-size 128 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code \
  --no-async-scheduling \
  --reasoning-parser poolside_v1 \
  --tool-call-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}'
Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.
At a glance

Checks that passed

  • Exact-tracking probes (17x20, 23x17, primes 50-70, sum of squares, letter count) x3: all correct with the routing fix; prose stays fluent but these collapse without it
  • Tool calls 4/4 non-streaming and 4/4 streaming; reasoning separated
  • Needle at 128K tokens: found (117 s cold)
  • LiveBench coding at temperature 1.0, 32K budget: LCB generation 85.9; coding completion 52.0 / 56.0 / 50.0 over three runs (the previous image scored 56.0 the same day)
  • Sampler check at temperature 1.0 (400 first-token draws x 4 prompts): sampled frequency within 0.01-0.02 of the model's probability

Notes

  • TP4, not TP8: at TP8 (one KV head per rank) the output was corrupt in lazy and compile mode.
  • Stay at 128K with 2048-token chunks: chunk x context beyond about 2048 x 130K trips the Synapse device watchdog ('No progress'), which kills the engine.
  • Sync scheduling (--no-async-scheduling) with TP 4 needs logits-empty-guard; async-inputs (in the profile) then has nothing to do.
  • rope-cpu-cache is not needed: the RoPE fix is upstream in v0.30.0, and the v0.30.0 profile leaves it out.
  • Do not use step-1 decode block buckets with VLLM_SKIP_WARMUP: every new context size compiles a graph (seconds to 40 s) and real sessions keep hitting new ones.
Guide

poolside's 118B mixture-of-experts coding and agent model (about 8B active; 256 experts, top 10, sigmoid router with a selection bias; sliding-window and full-attention layers), served in BF16 at 128K context on four Gaudi 2 cards.
Model poolside/Laguna-S-2.1
Hardware 4x Gaudi 2 (TP 4)
Image vllm-gaudi releases/v0.30.0 at 058f107, vLLM ced6857a, Gaudi SW 1.24.1
Precision BF16 (235 GB of weights)
Context 131,072 served (1,048,576 native)
Concurrency --max-num-seqs 16
Parsers tools and reasoning poolside_v1 (bundled with vLLM)
Recipe recipes/poolside/Laguna-S-2.1/recipe.yaml

LagunaForCausalLM is upstream in vLLM and runs on the generic HPU path: no model code to port. The routing fix below is the only model-specific change.

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.30.0, which also applies the release's other generic fixes (see Using a recipe).

Fix Why this model needs it
prompt-order prefix caching can read another request's KV in the same step
sampling-rows-refresh stale device sampling rows after a batch reorder
structured-output-rows structured-output steps sample each row with another request's parameters
so-row-owners with a prompt chunk that does not finish its prompt in the step, grammar masks and sampling parameters land one row off (another request forced into JSON, or an IndexError that stops the engine); without async scheduling such a chunk also leaked a stray token
bad-words-index bad_words raises IndexError in the engine core
logitsprocs-rows logit_bias, min_p and min_tokens are silently ignored
logits-empty-guard sync scheduling with TP 4: a prompt chunk that does not finish its prompt has no logits rows, and the tensor-parallel all-gather over the empty tensor stops the engine (ValidateSyncInputTensors)
gather-logprobs-int32 the logprob id corruption is still in v0.30 in lazy mode
logprobs-decode-guard a bad logprob id is replaced instead of stopping the API server
seeded-sampling a request with seed can segfault a TP worker in lazy mode
laguna-moe-sigmoid-routing vllm-gaudi's monolithic MoE path routes every non-grouped MoE with softmax and drops e_score_correction_bias, so it picks the wrong experts: prose stays fluent, counting, arithmetic and code collapse. The fix computes sigmoid scores, adds the bias for selection only and renormalizes, and fails closed
rank-exception-fatal an exception on a non-zero TP rank stops the engine instead of leaving ranks out of step

rope-cpu-cache is not listed: the RoPE fix is upstream in v0.30.0, and the v0.30.0 profile leaves it out.

On v0.30 the bundled poolside_v1 tool parser already accepts the template's <tool_call>NAME<arg_key> form without a newline, so no tool-parser fix is needed.

Verified

On the image above, four Gaudi 2 cards, the flags above:

  • Exact-tracking probes (17 x 20, 23 x 17, primes between 50 and 70, a sum of squares, a letter count), three runs each: all correct.
  • Tool calls 4/4 non-streaming and 4/4 streaming; reasoning separated from content.
  • Needle at 128K tokens: found (117 s cold).
  • LiveBench coding at temperature 1.0 with a 32K budget: LCB generation 85.9; coding completion 52.0, 56.0 and 50.0 over three runs. The previous image scored 56.0 the same day: run-to-run spread on this task is about 7 points.
  • Sampler check at temperature 1.0 (400 first-token draws x 4 prompts): sampled frequency of the top token within 0.01 to 0.02 of its probability.

Performance

Single measurements on four cards, the image above, warm shapes, unique prompts at temperature 0 with ignore_eos.

Test Result
1 stream, 1K prompt 53.2 tokens/s, TTFT 0.15 s
2 / 4 / 8 / 16 streams, per stream 56.2 / 44.7 / 36.7 / 23.2 tokens/s
4 / 8 / 16 streams, per replica 167 / 256 / 308 tokens/s
TTFT at 8K / 32K, cold prefix 1.52 to 1.61 / 7.79 s
TTFT at 8K, repeated (7,936 tokens cached) 0.52 s
Agent session (8 turns, about 1,700 new tokens each, after warm-up) TTFT 0.2 to 0.6 s, 44 to 48 tokens/s, 5.9 to 6.4 s per turn

Known issues

  • The first request at each new shape still compiles on a fresh replica (seconds to tens of seconds). With these coarse buckets the set fills quickly; with step-1 buckets it never does.
  • Laguna loops in its reasoning at temperature 0: at a 16K budget about 37% of coding answers hit the cap while still thinking. Use the model's sampling defaults and a generous max_tokens.

What not to do

  • Do not serve without laguna-moe-sigmoid-routing. The failure is silent: output reads well and exact answers are wrong, deterministically and at any precision.
  • Do not use TP 8. With one KV head per rank the output was corrupt in both lazy and compile mode; TP 4 (two KV heads per rank) is correct.
  • Do not go past 128K or raise --max-num-batched-tokens above 2,048. The Synapse device watchdog ("No progress") tracks chunk size times context: 2,048 x 130K passes, 2,048 x 212K and 4,096 x 96K kill the engine on all four cards.
  • Do not use step-1 decode block buckets (or the default prompt-context step of 1) with VLLM_SKIP_WARMUP=true: every new context length compiles a new graph. With step-1 buckets an 8-turn agent session ran at 29 tokens/s and 3.0 s TTFT on new context sizes, against 46 tokens/s and 1.2 s when repeated at sizes already compiled; real sessions keep hitting new sizes.