Gaudi Cookbook / Recipes
/

mistralai/Devstral-2-123B-Instruct-2512

123B dense agentic coding model, official FP8 checkpoint, 128K context with tool calling on four Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide
dense123B131,072 ctx servedfp8vllm-gaudi v0.26.0Text
Hardware
4 cards · TP 4
Image
vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1
Variant
The official FP8 checkpoint (128 GB), four Gaudi 2 cards.
Features
Fixes
profile v0.26.0
12 fixes this configuration depends on; opt-in fixes are named.
Launch
Shell 
export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.40
export VLLM_GRAPH_PROMPT_RATIO=0.30
export VLLM_GRAPH_PROMPT_STRATEGY=max_bs
export VLLM_GRAPH_DECODE_STRATEGY=max_bs
export VLLM_EXPONENTIAL_BUCKETING=False
export VLLM_PROMPT_CTX_BUCKET_MIN=0
export VLLM_PROMPT_CTX_BUCKET_STEP=32
export VLLM_PROMPT_CTX_BUCKET_MAX=992
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=4
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=4
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=2048
export VLLM_PROMPT_SEQ_BUCKET_MAX=4096
export VLLM_DECODE_BLOCK_BUCKET_MIN=128
export VLLM_DECODE_BLOCK_BUCKET_STEP=1024
export VLLM_DECODE_BLOCK_BUCKET_MAX=4096
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 || exit 1

vllm serve mistralai/Devstral-2-123B-Instruct-2512 \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.92 \
  --block-size 128 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --trust-remote-code
Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.
At a glance

Checks that passed

  • Known-answer prompts 8/8 at temperature 0 and 4/4 with 4 in flight
  • Tool calls: 5/5 valid calls sampled at 0.6, streamed call assembled, tool result used in the answer
  • top_logprobs 20, prompt_logprobs, JSON schema and logit_bias all take effect; server healthy after each
  • Needles, 3 depths each, cold: 3/3 at 32K and 3/3 at 120K tokens
  • Single stream 22.1 tokens/s, 4 streams 85.7 tokens/s aggregate (256 tokens, end to end)

Notes

  • Chunked prefill does not interleave a short request with a long prefill: the 4096-token budget goes to the long prompt's chunk every step, so a short request waits for it.
  • Licence: Modified MIT with a revenue limit (companies above USD 20 million monthly revenue need a commercial licence from Mistral AI).
Guide

Mistral AI's 123B dense agentic coding model from the official FP8 checkpoint, at 128K context with tool calling and prefix caching, on four Gaudi 2 cards.
Model mistralai/Devstral-2-123B-Instruct-2512
Licence Modified MIT: no rights for companies above USD 20 million monthly revenue without a commercial licence from Mistral AI
Hardware 4x Gaudi 2 (TP 4)
Image vllm-gaudi v0.26.0 (b059f9c), vLLM 568afb3a, Gaudi SW 1.24.1
Precision FP8 checkpoint (128 GB of weights)
Context 131,072 (the checkpoint allows 262,144)
Concurrency --max-num-seqs 4
Parsers tools mistral
Recipe recipes/mistralai/Devstral-2-123B-Instruct-2512/recipe.yaml

Required fixes

All from Fixes, all in --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe).

Fix Why this model needs it
prompt-order prefix caching is on: a request must not read KV that another request in the same step has not written yet
sampling-rows-refresh stale device sampling rows after a batch reorder
structured-output-rows the mistral tool parser gives every chat request a grammar, so every step that mixes a final prefill chunk with decodes sampled rows with another request's temperature, seed and logit_bias
bad-words-index bad_words raises IndexError in the engine core
logitsprocs-rows logit_bias, min_p and min_tokens are silently ignored
async-discard-counter AttributeError on the first preemption under async scheduling
resumed-prefill-logits a request resumed after preemption with prefix-cache hits repeats its earlier output
gather-logprobs-int32 top_logprobs token ids read back corrupted in lazy mode
prompt-logprobs prompt_logprobs requests crash or return wrong values
logprobs-decode-guard a bad logprob id is replaced instead of stopping the API server
seeded-sampling seeded requests can segfault a tensor-parallel worker in lazy mode
rank-exception-fatal an exception on a non-zero TP rank stops the engine instead of serving from out-of-step ranks

Verified

On vllm-gaudi v0.26.0, four Gaudi 2 cards, with exactly the launch command above in a container started from the stock image (the fixes applied at start-up). Requests went to the OpenAI-compatible API; needles used a different haystack per depth, so none was served from the prefix cache.

Check Result
Known-answer prompts (arithmetic, facts, translation), temperature 0 8/8
The same prompts, 4 in flight 4/4
Tool call, tool_choice: auto, sampled at 0.6 5/5 valid get_weather calls with the right argument
Streamed tool call name and JSON arguments assembled correctly
Tool result fed back the answer uses the returned values
top_logprobs: 20 20 entries on every token
prompt_logprobs: 5 returned; server healthy afterwards
response_format JSON schema valid object
logit_bias +100 on one token takes effect
Needle in a haystack, 3 depths, cold 3/3 at 32K tokens, 3/3 at 120K tokens
Server health after all checks healthy, no restarts

KV pool at start-up: 425,216 tokens.

Performance

Single runs on the server above (measured as: per-call, one server; times are end to end and include time to first token).

Test Result
One stream, short prompt, 256 tokens 22.1 tokens/s
4 streams, short prompts, 256 tokens each 85.7 tokens/s aggregate, 21.4 per stream
Cold prompt of 32K tokens (needle) 15 to 18 s to the answer
Cold prompt of 120K tokens (needle) 137 to 158 s to the answer

Known issues

  • Chunked prefill does not interleave a short request with a long prefill: the 4,096-token budget goes to the long prompt's chunk on every step, so a short request admitted behind it waits until that prefill finishes. Raising --max-num-seqs does not change this.
  • A cold 120K-token prompt took 137 to 158 s to answer (see Performance); size client and gateway timeouts for it. With prefix caching only the first turn of a conversation pays it.
  • A named tool_choice returns finish_reason: "stop", also when streaming.

What not to do

  • Do not switch to exponential bucketing for long-context traffic (see Buckets).
  • Do not go back to --max-num-seqs 1.