Gaudi Cookbook / Recipes
/

MiniMaxAI/MiniMax-M2.7

229B MoE (sigmoid routing, 256 experts) reasoning and agent model, official FP8 checkpoint, 128K context on four Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide
moe229B / 10B active131,072 ctx servedfp8vllm-gaudi v0.30.0Text
Hardware
4 cards · TP 4
Image
vllm-gaudi releases/v0.30.0 @ 058f107 · vLLM @ ced6857a · Gaudi SW 1.24.1
Variant
The official FP8 checkpoint (block 128 x 128, dynamic activations, 230 GB), four Gaudi 2 cards.
Features
Fixes
profile v0.30.0
12 fixes this configuration depends on; opt-in fixes are named.
Launch
Shell 
export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export PT_HPU_WEIGHT_SHARING=0
export VLLM_SKIP_WARMUP=true
export VLLM_GRAPH_RESERVED_MEM=0.2
export VLLM_GRAPH_PROMPT_RATIO=0.30
export VLLM_GRAPH_PROMPT_STRATEGY=max_bs
export VLLM_GRAPH_DECODE_STRATEGY=max_bs
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.30.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.30.0 || exit 1

vllm serve MiniMaxAI/MiniMax-M2.7 \
  --dtype auto \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.94 \
  --block-size 128 \
  --max-num-batched-tokens 1024 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice \
  --tool-call-parser minimax_m2 \
  --reasoning-parser minimax_m2
Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.
At a glance

Checks that passed

  • Known-answer prompts 7/8 at temperature 0 (one looped in its reasoning to max_tokens) and 4/4 with 4 in flight
  • 10 prompts with the checkpoint's sampling defaults (temperature 1.0, top_p 0.95, top_k 40): 10/10 finished; at temperature 0: 7/10
  • Tool calls: 5/5 valid calls sampled at 0.6, streamed call assembled, tool result used in the answer
  • top_logprobs 20, prompt_logprobs, JSON schema and logit_bias all take effect; server healthy after each
  • Needles, 3 depths each, cold: 3/3 at 32K and 3/3 at 120K tokens
  • Single stream 22.9 to 25.4 tokens/s, 4 streams 87.6 to 95.1 tokens/s aggregate (256 tokens, end to end)

Notes

  • Leave VLLM_ROW_PARALLEL_CHUNKS unset: on v0.26.0 any value above 1 stops vLLM at startup (AttributeError in register_hpu_lora_layers), and with 1024-token chunks a 16384-token chunking threshold would never be reached anyway.
  • Licence: non-commercial (MiniMax M2.7 licence); commercial use needs written authorization from MiniMax.
  • v0.26.0 cannot load this checkpoint: its minimax_m2 model calls FusedMoE.make_expert_params_mapping, which vLLM 568afb3a no longer has.
  • Loading is CPU-bound: every FP8 tensor is rescaled on the CPU, and vllm-gaudi sizes its load thread pool from the host's core count. Run without a CPU limit, or cap the pool at the quota.
Guide

MiniMax's 229B MoE reasoning and agent model (256 experts with sigmoid routing, about 10B active) from the official FP8 checkpoint, at 128K context with tool calling and prefix caching, on four Gaudi 2 cards.
Model MiniMaxAI/MiniMax-M2.7
Licence MiniMax M2.7 licence: non-commercial use; commercial use needs written authorization from MiniMax
Hardware 4x Gaudi 2 (TP 4)
Image vllm-gaudi v0.26.0 (b059f9c), vLLM 568afb3a, Gaudi SW 1.24.1
Precision FP8 checkpoint (block 128 x 128, dynamic activations, 230 GB)
Context 131,072 (the checkpoint allows 204,800)
Concurrency --max-num-seqs 4
Parsers tools minimax_m2, reasoning minimax_m2
Recipe recipes/MiniMaxAI/MiniMax-M2.7/recipe.yaml

Required fixes

All from Fixes, all in --profile v0.30.0, which also applies the release's other generic fixes (see Using a recipe).

Fix Why this model needs it
prompt-order prefix caching is on: a request must not read KV that another request in the same step has not written yet
sampling-rows-refresh stale device sampling rows after a batch reorder
structured-output-rows structured-output steps sample each row with another request's parameters
bad-words-index bad_words raises IndexError in the engine core
logitsprocs-rows logit_bias, min_p and min_tokens are silently ignored
resumed-prefill-logits a request resumed after preemption with prefix-cache hits repeats its earlier output
gather-logprobs-int32 top_logprobs token ids read back corrupted in lazy mode
prompt-logprobs prompt_logprobs requests crash or return wrong values
logprobs-decode-guard a bad logprob id is replaced instead of stopping the API server
seeded-sampling seeded requests can segfault a tensor-parallel worker in lazy mode
laguna-moe-sigmoid-routing MiniMax-M2 routes with sigmoid scores and a correction bias; the stock HPU MoE routing applies softmax and ignores the bias
rank-exception-fatal an exception on a non-zero TP rank stops the engine instead of serving from out-of-step ranks

Verified

On vllm-gaudi releases/v0.30.0 (058f107), four Gaudi 2 cards, with exactly the launch command above in a container started from the stock image (the fixes applied at start-up). Requests went to the OpenAI-compatible API; needles used a different haystack per depth, so none was served from the prefix cache.

Check Result
Known-answer prompts (arithmetic, facts, translation), temperature 0 7/8; the eighth repeated itself inside its reasoning until max_tokens (see Known issues)
The same prompts, 4 in flight 4/4
10 short prompts with the checkpoint's sampling defaults (temperature 1.0, top-p 0.95, top-k 40) 10/10 finished with a correct answer
The same 10 prompts at temperature 0, 8,192-token budget 7/10 finished; 3 looped in their reasoning
Tool call, tool_choice: auto, sampled at 0.6 5/5 valid get_weather calls with the right argument
Streamed tool call name and JSON arguments assembled correctly
Tool result fed back the answer uses the returned values
top_logprobs: 20 20 entries on every token
prompt_logprobs: 5 returned; server healthy afterwards
response_format JSON schema valid object
logit_bias +100 on one token takes effect
Needle in a haystack, 3 depths, cold 3/3 at 32K tokens, 3/3 at 120K tokens
Server health after all checks healthy, no restarts

KV pool at start-up: 481,152 tokens (3.67 requests of 131,072 tokens).

Performance

Single runs on the server above (measured as: per-call, one server; times are end to end and include time to first token and the model's reasoning).

Test Result
One stream, short prompt, 256 tokens 22.9 to 25.4 tokens/s (two runs)
4 streams, short prompts, 256 tokens each 87.6 to 95.1 tokens/s aggregate (two runs)
Cold prompt of 32K tokens (needle), answer after reasoning 53 to 56 s (213 s for the first, which also compiled its graphs)
Cold prompt of 120K tokens (needle), answer after reasoning 144 to 226 s

Known issues

  • At temperature 0 the model sometimes repeats itself inside its reasoning until max_tokens (3 of 10 short prompts here). Use the checkpoint's sampling defaults (temperature 1.0, top-p 0.95, top-k 40); vLLM applies them from generation_config.json when a request does not set them.
  • Loading is CPU-bound: every FP8 tensor is rescaled on the CPU, and vllm-gaudi sizes its load thread pool from the host's core count. Run without a CPU limit, or cap the pool at the quota.
  • A full warm-up of this MoE takes far too long; graphs compile on first use, so the first request at each new shape is slow.

What not to do

  • Do not set VLLM_ROW_PARALLEL_CHUNKS above 1: on v0.26.0 any such value stops vLLM at startup (AttributeError in register_hpu_lora_layers), and with 1,024-token chunks a 16,384-token chunking threshold would never be reached anyway.
  • Do not use the v0.26.0 release for this model (see above).