Gaudi Recipes

Pick a model, copy the vllm serve line that runs on Intel Gaudi 2. Every recipe uses a released vllm-gaudi image and a public checkpoint, lists the fixes it needs, and shows the checks that passed. Build an image →

38 recipes38 verified 15 providersGaudi SW 1.24.1 vllm-gaudi v0.26.0 · v0.30.0

New kernel

Up to 2.5× faster MXFP4 MoE decode on Gaudi 2

Custom TPC kernels run the routed experts straight from the packed MXFP4 weights, with no bf16 unpack round trip. On one 8-card node they took GLM-5.3 from 23.3 to 59.1 tok/s per stream and Kimi-K2.7 from 17.9 to 44.5, with lower numerical error than the stock op, and also serve DeepSeek-V4.1-Flash, GLM-5.2, Muse Spark Open and MiMo-V2.6-Pro. Time to first token is unchanged, except on GLM-5.x, where a prefill unpack kernel cuts it by 10% to 19%.

Every model above passes its recipe's gates with the kernel on: tool calls, long-context needles, and GSM8K or perplexity against a reference.

Read the write-up GLM-5.3 recipe Source

2.5×

GLM-5.3 single stream: 23.3 → 59.1 tok/s on one 8-card node

2.5×

Kimi-K2.7 single stream at 16K context: 17.9 → 44.5 tok/s, same pod

2.4×

Muse Spark Open, 4 streams aggregate: 43.3 → 102.8 tok/s, same pod

2–4×

lower error than the stock op against an fp32 reference

Model	Setup	Stock op	With the kernel	Gain	Measured as
GLM-5.3 (MXFP4)	8 cards, TP 8 + EP 8, 1 stream	23.3 tok/s	59.1	2.5×	3 same-node A/Bs chained	recipe
Kimi-K2.7-Code (MXFP4)	8 cards, TP 8 + EP 8, 16K, 1 stream	17.9 tok/s	44.5	2.5×	same-pod A/B, published recipe	recipe
Muse Spark Open	8 cards, 1 stream	22.7 tok/s	40.8	1.8×	same-pod A/B, published recipe	recipe
Muse Spark Open	8 cards, 4 streams, aggregate	43.3 tok/s	102.8	2.4×	same-pod A/B, published recipe	recipe
DeepSeek-V4.1-Flash (MXFP4)	8 cards, TP 8, 1 stream	20.7 tok/s	37.6	1.8×	same-pod A/B, published recipe	recipe
GLM-5.2 (MXFP4)	8 cards, 1 stream	23.1 tok/s	47.8	2.1×	stock pod vs kernel pods	recipe
MiMo-V2.6-Pro (MXFP4)	8 cards, 1 stream	18.1 tok/s	39.7	2.2×	same node: stock pod, then kernel pod	recipe
openai/gpt-oss-120b	2 cards, TP 2 + EP, 4 in flight	16.6 tok/s per stream	52.4	3.2×	same-node A/B	recipe

GLM-5.x, DeepSeek-V4.1-Flash, Kimi-K2.7, Muse Spark Open and MiMo-V2.6-Pro run the kernel through their ports, each with a recipe here. GLM-5.3 reaches 69.6 tok/s single stream with decode-step and expert-placement work on top of the kernel.

All recipes38 shown

744B/41Bmxfp4128K ctx8× Gaudi 2TextPort

744B MoE (about 41B active) with DeepSeek sparse attention, AMD's MXFP4 release repacked byte for byte and served at 128K context with MTP on one 8-card Gaudi 2 node, through the glm-5-mxfp4 port.

Kimi-K2.7-Code-MXFP4

1T/32Bmxfp4128K ctx8× Gaudi 2TextPort

1T-parameter MoE coding model (32B active, MLA, 384 experts) served text-only from AMD's MXFP4 release at 128K context on one 8-card Gaudi 2 node, through the kimi-k2-7 port, with the MXFP4 decode kernels behind one switch.

MiniMax-M3-MXFP4

427B/23Bfp8128K ctx8× Gaudi 2TextPort

427B / 23B-active MoE served text-only on one 8-card Gaudi 2 node at 128K context, from AMD's MXFP4 checkpoint re-encoded exactly to per-channel FP8 experts.

Muse-Spark-Open-816B-A42B-Quark-MXFP4

816B/42Bmxfp4256K ctx8× Gaudi 2TextPort

816B MoE (42B active) with a latent MoE and per-expert output norm, the AMD Quark MXFP4 checkpoint as published, 256K tokens per request on one 8-card Gaudi 2 node, text only, through the muse-spark-open port.

DeepSeek-V4.1-Flash-1017

749B/17Bmxfp4128K ctx8× Gaudi 2TextPort

40-layer MoE (384 routed experts, top-6) with compressed sparse attention, hyper-connections and Engram tables, served at 128K context on one 8-card Gaudi 2 node through the deepseek-v4-1-flash port, from a lossless re-layout of the public checkpoint.

31Bbf16128K ctx4× Gaudi 2Multimodal

31B dense multimodal (image + text) instruct model, BF16, 128K context on four Gaudi 2 cards.

5.1Bbf1632K ctx1× Gaudi 2Text

Small Gemma 4 instruct model (2.3B effective, 20 KV-shared layers) with KV-sharing fast prefill, BF16, text only, on one Gaudi 2 card.

Llama-4-Scout-17B-16E-Instruct

109B/17Bbf16128K ctx4× Gaudi 2Multimodal

109B MoE (17B active, 16 experts, top-1) multimodal instruct model, BF16, 128K context on four Gaudi 2 cards.

Muse-Glimmer-30B

29.6Bbf16128K ctx2× Gaudi 2MultimodalPort

29.6B dense vision-language agent model (3:1 sliding/global attention, image input), BF16, 128K context on two Gaudi 2 cards.

229B/10Bfp8128K ctx4× Gaudi 2Text

229B MoE (sigmoid routing, 256 experts) reasoning and agent model, official FP8 checkpoint, 128K context on four Gaudi 2 cards.

Devstral-2-123B-Instruct-2512

123Bfp8128K ctx4× Gaudi 2Text

123B dense agentic coding model, official FP8 checkpoint, 128K context with tool calling on four Gaudi 2 cards.

117B/5.1Bmxfp4128K ctx4× Gaudi 2Text2.96× with kernel

117B MoE (5.1B active) reasoning model in native MXFP4, harmony format, 128K context on four Gaudi 2 cards.

35B/3Bbf16256K ctx2× Gaudi 2Multimodal

35B hybrid MoE (Gated DeltaNet + attention, 3B active) agentic coding model with vision, BF16, native 256K context on two Gaudi 2 cards.

118B/8Bbf16128K ctx4× Gaudi 2Text

118B MoE (8B active) coding and agent model with reasoning, BF16, 128K context on four Gaudi 2 cards.

Qwen-AgentWorld-35B-A3B

35B/3Bbf16256K ctx2× Gaudi 2Text

35B hybrid MoE (Gated DeltaNet + attention, 3B active) agent model, BF16, native 256K context on two Gaudi 2 cards.

Qwen3-235B-A22B-Instruct-2507-FP8

235B/22Bfp8128K ctx4× Gaudi 2Text

235B MoE (22B active) instruct model, FP8 block-quantized checkpoint, 128K context on four Gaudi 2 cards.

Qwen3-235B-A22B-Thinking-2507-FP8

235B/22Bfp8128K ctx4× Gaudi 2Text

235B MoE (22B active) reasoning model, FP8 block-quantized checkpoint, 128K context on four Gaudi 2 cards.

Qwen3-30B-A3B-Instruct-2507

30.5B/3.3Bbf16128K ctx1× Gaudi 2Text

30B MoE (3B active) instruct model, non-thinking, served at 128K context on one Gaudi 2 card in BF16.

Qwen3-30B-A3B-Thinking-2507

30.5B/3.3Bbf16128K ctx1× Gaudi 2Text

30B MoE (3B active) reasoning model, always thinks, served at 128K context on one Gaudi 2 card in BF16.

Qwen3-Coder-30B-A3B-Instruct

30.5B/3.3Bbf16128K ctx1× Gaudi 2Text

30B MoE (3B active) coding model with tool calling, served at 128K context on one Gaudi 2 card in BF16.

Qwen3-Coder-Next

80B/3Bbf16256K ctx4× Gaudi 2Text

80B hybrid MoE (Gated DeltaNet + attention, 3B active) coding model with tool calling, BF16, native 256K context on four Gaudi 2 cards.

Qwen3-Embedding-4B

4Bbf1632K ctx1× Gaudi 2Embedding

4B instruction-aware text embedding model (2560 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.

Qwen3-Embedding-8B

8Bbf1632K ctx1× Gaudi 2Embedding

8B instruction-aware text embedding model (4096 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.

Qwen3-VL-32B-Instruct-FP8

32Bfp8128K ctx2× Gaudi 2Multimodal

32B dense vision-language model (images and video) in FP8, 128K context on two Gaudi 2 cards.

Qwen3-VL-Embedding-8B

8Bbf1632K ctx1× Gaudi 2EmbeddingMultimodal

8B multimodal embedding model (text, images, screenshots; 4096 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.

Qwen3.5-122B-A10B-FP8

122B/10Bfp8256K ctx4× Gaudi 2Text

122B hybrid MoE (Gated DeltaNet + attention, 10B active) reasoning model, FP8, native 256K context on four Gaudi 2 cards.

Qwen3.5-27B-FP8

27Bfp8256K ctx1× Gaudi 2Multimodal

27B dense hybrid (Gated DeltaNet + attention) reasoning model with vision, FP8, native 256K context on one Gaudi 2 card.

Qwen3.5-397B-A17B-FP8

397B/17Bfp8256K ctx8× Gaudi 2Text

397B MoE (17B active) hybrid (Gated DeltaNet + attention) reasoning model, FP8, native 256K context, text only, on one 8-card Gaudi 2 node.

Qwen3.6-27B-FP8

27Bfp8256K ctx1× Gaudi 2Multimodal

27B dense hybrid (Gated DeltaNet + attention) reasoning model with vision, FP8, native 256K context on one Gaudi 2 card.

Qwen3.6-35B-A3B-FP8

35B/3Bfp8256K ctx1× Gaudi 2Multimodal

35B MoE (3B active) hybrid (Gated DeltaNet + attention) reasoning model with vision, FP8, native 256K context on one Gaudi 2 card.

27Bbf16256K ctx1× Gaudi 2Multimodal

27B dense hybrid (Gated DeltaNet + attention) reasoning model with vision, BF16, native 256K context on one Gaudi 2 card, 4 sequences.

Beam-1.0-501B-A23B-FP8

501B/23Bfp8256K ctx8× Gaudi 2TextPort

501B MoE (23B active) reasoning and coding model, compressed-tensors FP8, 256K context on one 8-card Gaudi 2 node, through the beam-1-0 port.

thinkingmachines

276B/12Bbf16256K ctx8× Gaudi 2TextPort

276B-A12B MoE with relative-position attention and short convs, text only, BF16 at 256K context on one 8-card Gaudi 2 node (port).

MiMo-V2.6-Flash-RL

309B/15Bfp8512K ctx8× Gaudi 2TextPort

309B MoE (15B active) with 128-token sliding-window attention and sinks, served at 512K context on one 8-card Gaudi 2 node through a port.

MiMo-V2.6-Pro-RL

1.02T/42Bmxfp4256K ctx8× Gaudi 2TextPort

1.02T MoE (42B active), 128-token sliding windows with sinks, MXFP4 experts on the MXFP4 decode kernels, 256K context on one 8-card node.

106B/12Bfp832K ctx2× Gaudi 2Multimodal

106B MoE vision-language model (12B active) in FP8, 32K context on two Gaudi 2 cards.

744B/41Bmxfp4128K ctx8× Gaudi 2TextPort

744B MoE (about 41B active) with DeepSeek sparse attention, quantized to MXFP4 experts from the BF16 release and served at 128K context with MTP on one 8-card Gaudi 2 node, through the glm-5-mxfp4 port.

320B/18Bfp8256K ctx8× Gaudi 2TextMultimodalPort

320B MoE (18B active) with KDA linear attention, NoPE MLA, mHC and image input, block FP8, 256K context on eight Gaudi 2 cards through the glm-5-3-flash port.

Ports and resultsAll ports → · All results →

Muse Spark Open

816B MoE, MXFP4, 8 cards

Needles pass at 256K, 512K and 1M tokens

perplexity within 0.4% of an fp32 reference

KDA + DSA + mHC MoE, 8 cards

22 to 24 tok/s from 2K to 517K context

decode flat with context, same image and code

DeepSeek-V4.1-Flash

New attention and experts, 8 cards

23.6 → 35.0 tok/s decode

same-node A/B; prefill 1.35 to 1.55x

GLM-5.2 and GLM-5.3 MXFP4

744B MoE, sparse DSA, 8 cards

23.25 → 45.6 tok/s, first kernel

same-pod A/B; needles fixed to 128K

501B MoE, 256K, 8 cards

Weight load 2177 → 356 s

same-node A/B, bit-identical state

MiMo-V2.6-Flash

Sliding window with sinks, 512K

5.33x the KV token capacity

identical outputs; needles 3/3 at 500K

1T MoE, TP 16 over two nodes

FP8 experts about 3x MXFP4 at 8 streams

different conditions, named on the page

Platform findings

NIC stalls, FusedSDPA, fp32 trig, sampling

7.56 → 15.45 tok/s from one NIC fix

Prefix caching on GDN hybrids

Qwen3.5 / 3.6 / 3.8 style models

Later-turn TTFT 19.8 → 2.6 s

same-pod A/B, agent replay

Anatomy of a MoE decode step

8-rank critical path, EP layouts

51.6 → 57.8 tok/s, bit-identical

Two-node pipeline parallel

PP 2 x TP 8 over the host NIC

709,888-token KV pool

needles pass at 637K tokens

Gemma-4-E2B fast prefill

KV-shared layers skip prefill work

72.8k → 169.7k tok/s prefill

same card, token-exact

Fast text-to-speech

Qwen3-TTS, Chatterbox, Orpheus

Chatterbox Turbo 10 to 13x real time

per-call, one card

EmbeddingGemma 2

Bidirectional multimodal embedder

95k to 235k text tok/s per card

per-call, one card; cosine ≥ 0.9995

Browse by provider

Thinking Machines

thinkingmachines

More in the cookbook

Fail-closed runtime patches for bugs in released images, each tied to an upstream pull request.

MXFP4 MoE decode TPC kernels: up to 2.5x faster decode on GLM-5.x, Kimi-K2.7, Muse Spark Open and DeepSeek-V4.1.

Pod spec, bin-pack scheduling, warm-up, health, rollouts, autoscaling and routing.

Use with an agent

Point your coding agent at the repository; AGENTS.md tells it where everything is.