# Gaudi Cookbook

> Tested recipes, runtime fixes, kernels and fleet practices for serving open LLMs with vLLM on Intel Gaudi 2 through the vllm-gaudi plugin. Gaudi software 1.24.1, vllm-gaudi v0.26.0 and releases/v0.30.0, lazy mode unless a recipe sets compile mode.

Agents: read AGENTS.md first. It maps tasks to files and gives the rules: check the image before patching, keep verified and untested apart, and ask before touching shared hardware. The links below point at raw Markdown and YAML.

## Start here

- [AGENTS.md](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/AGENTS.md): where to look for each task, how to turn a recipe into a command, and the rules.
- [Getting started](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/getting-started.md): software baseline, image build steps, container flags.
- [Recipe format](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/README.md): the recipe.yaml schema and its gaudi block.
- [Fixes contract](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/README.md): how the fix pack applies, profiles, exit codes, adding a fix.
- [Port contract](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/ports/README.md): how a model port is laid out and built from a stock image.

## Recipes

- [Recipe index](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/index.md): every model with cards, precision, image, features and status.
- [Qwen3-30B-A3B-2507 (Instruct and Thinking)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-30b-a3b-2507.md): Qwen's 30B mixture-of-experts model (3.3B active), in its July 2025 non-thinking (Instruct) and always-thinking (Thinking) variants, served in BF16 at 128K context on a single Gaudi 2 card.
  - [Qwen/Qwen3-30B-A3B-Instruct-2507 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-30B-A3B-Instruct-2507/recipe.yaml)
  - [Qwen/Qwen3-30B-A3B-Thinking-2507 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-30B-A3B-Thinking-2507/recipe.yaml)
- [Qwen3-Coder-30B-A3B-Instruct](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-coder-30b-a3b-instruct.md): 30B MoE (3B active) coding model with tool calling, served at 128K context on one Gaudi 2 card in BF16.
  - [Qwen/Qwen3-Coder-30B-A3B-Instruct recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-Coder-30B-A3B-Instruct/recipe.yaml)
- [Qwen3.5-27B-FP8](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-5-27b-fp8.md): 27B dense hybrid (Gated DeltaNet + attention) reasoning model with vision, FP8, native 256K context on one Gaudi 2 card.
  - [Qwen/Qwen3.5-27B-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3.5-27B-FP8/recipe.yaml)
- [Qwen3.8-27B](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-8-27b.md): 27B dense hybrid (Gated DeltaNet + attention) reasoning model with vision, BF16, native 256K context on one Gaudi 2 card, 4 sequences.
  - [Qwen/Qwen3.8-27B recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3.8-27B/recipe.yaml)
- [Qwen3.6-27B-FP8](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-6-27b-fp8.md): 27B dense hybrid (Gated DeltaNet + attention) reasoning model with vision, FP8, native 256K context on one Gaudi 2 card.
  - [Qwen/Qwen3.6-27B-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3.6-27B-FP8/recipe.yaml)
- [Qwen3.6-35B-A3B-FP8](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-6-35b-a3b-fp8.md): 35B MoE (3B active) hybrid (Gated DeltaNet + attention) reasoning model with vision, FP8, native 256K context on one Gaudi 2 card.
  - [Qwen/Qwen3.6-35B-A3B-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3.6-35B-A3B-FP8/recipe.yaml)
- [Qwen3-Embedding-8B](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-embedding-8b.md): 8B instruction-aware text embedding model (4096 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.
  - [Qwen/Qwen3-Embedding-8B recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-Embedding-8B/recipe.yaml)
- [Qwen3-Embedding-4B](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-embedding-4b.md): 4B instruction-aware text embedding model (2560 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.
  - [Qwen/Qwen3-Embedding-4B recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-Embedding-4B/recipe.yaml)
- [Qwen3-VL-Embedding-8B](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-vl-embedding-8b.md): 8B multimodal embedding model (text, images, screenshots; 4096 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.
  - [Qwen/Qwen3-VL-Embedding-8B recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-VL-Embedding-8B/recipe.yaml)
- [Gemma 4 E2B IT](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/gemma-4-e2b-it.md): Small Gemma 4 instruct model (2.3B effective, 20 KV-shared layers) with KV-sharing fast prefill, BF16, text only, on one Gaudi 2 card.
  - [google/gemma-4-E2B-it recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/google/gemma-4-E2B-it/recipe.yaml)
- [Qwen-AgentWorld-35B-A3B](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen-agentworld-35b-a3b.md): 35B hybrid MoE (Gated DeltaNet + attention, 3B active) agent model, BF16, native 256K context on two Gaudi 2 cards.
  - [Qwen/Qwen-AgentWorld-35B-A3B recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen-AgentWorld-35B-A3B/recipe.yaml)
- [Ornith-1.0-35B](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/ornith-1-0-35b.md): 35B hybrid MoE (Gated DeltaNet + attention, 3B active) agentic coding model with vision, BF16, native 256K context on two Gaudi 2 cards.
  - [ornith-ai/Ornith-1.0-35B recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/ornith-ai/Ornith-1.0-35B/recipe.yaml)
- [Qwen3-VL-32B-Instruct-FP8](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-vl-32b-instruct-fp8.md): 32B dense vision-language model (images and video) in FP8, 128K context on two Gaudi 2 cards.
  - [Qwen/Qwen3-VL-32B-Instruct-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-VL-32B-Instruct-FP8/recipe.yaml)
- [GLM-4.5V (FP8)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/glm-4-5v-fp8.md): 106B MoE vision-language model (12B active) in FP8, 32K context on two Gaudi 2 cards.
  - [zai-org/GLM-4.5V-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/zai-org/GLM-4.5V-FP8/recipe.yaml)
- [Muse Glimmer 30B](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/muse-glimmer-30b.md): 29.6B dense vision-language agent model (3:1 sliding/global attention, image input), BF16, 128K context on two Gaudi 2 cards.
  - [meta-models/Muse-Glimmer-30B recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/meta-models/Muse-Glimmer-30B/recipe.yaml)
- [Qwen3-235B-A22B-2507-FP8 (Instruct and Thinking)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-235b-a22b-2507-fp8.md): Qwen's 235B mixture-of-experts model (22B active, 128 experts, top 8) in its July 2025 Instruct and Thinking variants, from the official FP8 block-quantized checkpoints, at 128K context on four Gaudi 2 cards.
  - [Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8/recipe.yaml)
  - [Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-235B-A22B-Thinking-2507-FP8/recipe.yaml)
- [Qwen3.5-122B-A10B-FP8](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-5-122b-a10b-fp8.md): 122B hybrid MoE (Gated DeltaNet + attention, 10B active) reasoning model, FP8, native 256K context on four Gaudi 2 cards.
  - [Qwen/Qwen3.5-122B-A10B-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3.5-122B-A10B-FP8/recipe.yaml)
- [Qwen3-Coder-Next](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-coder-next.md): 80B hybrid MoE (Gated DeltaNet + attention, 3B active) coding model with tool calling, BF16, native 256K context on four Gaudi 2 cards.
  - [Qwen/Qwen3-Coder-Next recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3-Coder-Next/recipe.yaml)
- [Devstral-2-123B-Instruct-2512](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/devstral-2-123b-instruct-2512.md): 123B dense agentic coding model, official FP8 checkpoint, 128K context with tool calling on four Gaudi 2 cards.
  - [mistralai/Devstral-2-123B-Instruct-2512 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/mistralai/Devstral-2-123B-Instruct-2512/recipe.yaml)
- [Gemma 4 31B IT](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/gemma-4-31b-it.md): 31B dense multimodal (image + text) instruct model, BF16, 128K context on four Gaudi 2 cards.
  - [google/gemma-4-31B-it recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/google/gemma-4-31B-it/recipe.yaml)
- [gpt-oss-120b](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/gpt-oss-120b.md): 117B MoE (5.1B active) reasoning model in native MXFP4, harmony format, 128K context on four Gaudi 2 cards.
  - [openai/gpt-oss-120b recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/openai/gpt-oss-120b/recipe.yaml)
- [Llama 4 Scout 17B-16E Instruct](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/llama-4-scout-17b-16e-instruct.md): 109B MoE (17B active, 16 experts, top-1) multimodal instruct model, BF16, 128K context on four Gaudi 2 cards.
  - [meta-llama/Llama-4-Scout-17B-16E-Instruct recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/meta-llama/Llama-4-Scout-17B-16E-Instruct/recipe.yaml)
- [Laguna-S-2.1](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/laguna-s-2-1.md): 118B MoE (8B active) coding and agent model with reasoning, BF16, 128K context on four Gaudi 2 cards.
  - [poolside/Laguna-S-2.1 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/poolside/Laguna-S-2.1/recipe.yaml)
- [MiniMax-M2.7](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/minimax-m2-7.md): 229B MoE (sigmoid routing, 256 experts) reasoning and agent model, official FP8 checkpoint, 128K context on four Gaudi 2 cards.
  - [MiniMaxAI/MiniMax-M2.7 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/MiniMaxAI/MiniMax-M2.7/recipe.yaml)
- [Qwen3.5-397B-A17B-FP8](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/qwen3-5-397b-a17b-fp8.md): 397B MoE (17B active) hybrid (Gated DeltaNet + attention) reasoning model, FP8, native 256K context, text only, on one 8-card Gaudi 2 node.
  - [Qwen/Qwen3.5-397B-A17B-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/Qwen/Qwen3.5-397B-A17B-FP8/recipe.yaml)
- [Beam-1.0-501B-A23B-FP8](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/beam-1-0-501b-a23b-fp8.md): 501B MoE (23B active) reasoning and coding model, compressed-tensors FP8, 256K context on one 8-card Gaudi 2 node, through the beam-1-0 port.
  - [reflection/Beam-1.0-501B-A23B-FP8 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/reflection/Beam-1.0-501B-A23B-FP8/recipe.yaml)
- [Muse Spark Open (MXFP4)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/muse-spark-open.md): 816B MoE (42B active) with a latent MoE and per-expert output norm, the AMD Quark MXFP4 checkpoint as published, 256K tokens per request on one 8-card Gaudi 2 node, text only, through the muse-spark-open port.
  - [amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4/recipe.yaml)
- [GLM-5.2 (MXFP4)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/glm-5-2-mxfp4.md): 744B MoE (about 41B active) with DeepSeek sparse attention, AMD's MXFP4 release repacked byte for byte and served at 128K context with MTP on one 8-card Gaudi 2 node, through the glm-5-mxfp4 port.
  - [amd/GLM-5.2-MXFP4 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/amd/GLM-5.2-MXFP4/recipe.yaml)
- [GLM-5.3 (MXFP4)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/glm-5-3-mxfp4.md): 744B MoE (about 41B active) with DeepSeek sparse attention, quantized to MXFP4 experts from the BF16 release and served at 128K context with MTP on one 8-card Gaudi 2 node, through the glm-5-mxfp4 port.
  - [zai-org/GLM-5.3-BF16 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/zai-org/GLM-5.3-BF16/recipe.yaml)
- [GLM-5.3-Flash](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/glm-5-3-flash.md): 320B MoE (18B active) with KDA linear attention, NoPE MLA, mHC and image input, block FP8, 256K context on eight Gaudi 2 cards through the glm-5-3-flash port.
  - [zai-org/GLM-5.3-Flash recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/zai-org/GLM-5.3-Flash/recipe.yaml)
- [DeepSeek-V4.1-Flash-1017](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/deepseek-v4-1-flash-1017.md): 40-layer MoE (384 routed experts, top-6) with compressed sparse attention, hyper-connections and Engram tables, served at 128K context on one 8-card Gaudi 2 node through the deepseek-v4-1-flash port, from a lossless re-layout of the public checkpoint.
  - [deepseek-ai/DeepSeek-V4.1-Flash-1017 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/deepseek-ai/DeepSeek-V4.1-Flash-1017/recipe.yaml)
- [Kimi-K2.7-Code (MXFP4)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/kimi-k2-7-code.md): 1T-parameter MoE coding model (32B active, MLA, 384 experts) served text-only from AMD's MXFP4 release at 128K context on one 8-card Gaudi 2 node, through the kimi-k2-7 port, with the MXFP4 decode kernels behind one switch.
  - [amd/Kimi-K2.7-Code-MXFP4 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/amd/Kimi-K2.7-Code-MXFP4/recipe.yaml)
- [MiniMax-M3 (FP8 experts)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/minimax-m3.md): 427B / 23B-active MoE served text-only on one 8-card Gaudi 2 node at 128K context, from AMD's MXFP4 checkpoint re-encoded exactly to per-channel FP8 experts.
  - [amd/MiniMax-M3-MXFP4 recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/amd/MiniMax-M3-MXFP4/recipe.yaml)
- [MiMo-V2.6-Flash-RL](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/mimo-v2-6-flash-rl.md): 309B MoE (15B active) with 128-token sliding-window attention and sinks, served at 512K context on one 8-card Gaudi 2 node through a port.
  - [XiaomiMiMo/MiMo-V2.6-Flash-RL recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/XiaomiMiMo/MiMo-V2.6-Flash-RL/recipe.yaml)
- [MiMo-V2.6-Pro-RL](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/mimo-v2-6-pro-rl.md): 1.02T MoE (42B active), 128-token sliding windows with sinks, MXFP4 experts on the MXFP4 decode kernels, 256K context on one 8-card node.
  - [XiaomiMiMo/MiMo-V2.6-Pro-RL recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/XiaomiMiMo/MiMo-V2.6-Pro-RL/recipe.yaml)
- [Inkling-Small](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/recipes/inkling-small.md): 276B-A12B MoE with relative-position attention and short convs, text only, BF16 at 256K context on one 8-card Gaudi 2 node (port).
  - [thinkingmachines/Inkling-Small recipe.yaml](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/recipes/thinkingmachines/Inkling-Small/recipe.yaml)

## Fixes

- [Fix index](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/fixes/index.md): status table with affected setups, releases, profiles and upstream links.
- [prompt-order](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/prompt-order/README.md): Run a step's prompts in order of context length, so a prefix-cache hit is never read before the request that computes it has written it. Affects: every model served with prefix caching (the default).
- [sampling-rows-refresh](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/sampling-rows-refresh/README.md): Re-copy the device-side sampling parameters after the input batch reorders its rows, so every request is sampled with its own temperature, top-p, top-k, min-p and penalties. Affects: every model, whenever concurrent requests use different sampling parameters.
- [structured-output-rows](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/structured-output-rows/README.md): Sample each row of a structured-output step with its own request's sampling parameters. Affects: every model, in any step that has a request with a grammar.
- [so-row-owners](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/so-row-owners/README.md): Give grammar masks, sampling parameters and logprobs to the logits rows that actually exist in a structured-output step. Affects: every model, in steps with a grammar request and a partial prompt chunk.
- [bad-words-index](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/bad-words-index/README.md): Key `bad_words` to the rows that are actually being sampled, so one request's `bad_words` can neither stop the engine nor ban tokens on another request's output. Affects: every model, whenever a request sets `bad_words`.
- [logitsprocs-rows](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/logitsprocs-rows/README.md): Make the `min_p`, `logit_bias` and `min_tokens` request options take effect on HPU. Affects: every model, whenever a request sets `min_p`, `logit_bias` or `min_tokens`.
- [async-inputs](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/async-inputs/README.md): Seed the device decode input ids from the host before the async-scheduling scatter, so a decode request that sat out a step does not read a stale token. Affects: every model served with async scheduling.
- [async-discard-counter](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/async-discard-counter/README.md): Make the HPU async scheduler use vLLM's `async_tokens_to_discard` counter, so a preempted request no longer stops the engine with an `AttributeError`. Affects: every model served with async scheduling (the default) when a request is preempted.
- [resumed-prefill-logits](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/resumed-prefill-logits/README.md): Sample one token, not one per recomputed position, when a request resumes after preemption with prefix-cache hits. Affects: every model served with prefix caching, when requests are preempted.
- [hybrid-decode-pad](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/hybrid-decode-pad/README.md): Keep decode padding out of live KV blocks on hybrid models whose attention KV block is split into smaller kernel blocks. Affects: hybrid models (GDN, linear attention or Mamba next to attention), e.g. Qwen3.5 / Qwen3-Next class.
- [gdn-prefix-cache](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/gdn-prefix-cache/README.md): Make prefix caching correct on Gated DeltaNet hybrids (Qwen3.5, Qwen3.6, Qwen3-Next class) while keeping the full KV pool: each cached block's GDN state is stored as a checkpoint inside the attention pages of that block. Affects: GDN hybrids served with `--enable-prefix-caching` (compact GDN, the default).
- [penalty-shapes](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/penalty-shapes/README.md): Pad the token tensors used by repetition, presence and frequency penalties to bucketed lengths, so a batch that uses penalties stops recompiling HPU graphs on every decode step. Affects: requests that set `repetition_penalty`, `presence_penalty` or `frequency_penalty` (performance).
- [logits-empty-guard](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/logits-empty-guard/README.md): Skip the tensor-parallel logits gather when a step has no logits rows, so long prompts do not stop the engine. Affects: every model served with tensor parallelism and sync scheduling (`--no-async-scheduling`).
- [gather-logprobs-int32](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/gather-logprobs-int32/README.md): Cast logprob token ids to int32 before concatenating them, so `top_logprobs` ids read back correctly on HPU. Affects: every model, requests with `logprobs` / `top_logprobs` (and `prompt_logprobs`).
- [prompt-logprobs](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/prompt-logprobs/README.md): Make `prompt_logprobs` safe and correct in the HPU model runner: no engine crash on models with a flat hidden-state output, correct values, bounded memory. Affects: every model, requests with `prompt_logprobs` (or `/v1/completions` with `echo` and `logprobs`).
- [logprobs-decode-guard](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/logprobs-decode-guard/README.md): Stop one undecodable logprob token id from shutting down the API server. Affects: every model, requests with `logprobs` / `top_logprobs`.
- [seeded-sampling](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/seeded-sampling/README.md): Seeded random sampling on HPU without `torch.Generator` ops on the device, so a request with a `seed` cannot crash a worker, and seeded output does not depend on the batch. Affects: every model, requests with `seed` and temperature > 0.
- [topk-subset-sampler](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/topk-subset-sampler/README.md): Top-k / top-p sampling on HPU without sorting the whole vocabulary on every decode step. Affects: every model, requests with `top_k` (with or without `top_p`), including defaults from `generation_config.json`.
- [sink-once](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/sink-once/README.md): Count attention sinks once per sequence in paged decode, and apply them in FusedSDPA prefill on v0.26.0. Affects: models with attention sinks (for example gpt-oss).
- [swa-decode-window](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/swa-decode-window/README.md): Make the sliding window exact in paged decode for gpt-oss, instead of 129 to 256 visible tokens for a 128-token window. Affects: gpt-oss (20B and 120B).
- [moe-explicit-overload](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/moe-explicit-overload/README.md): Call the per-channel FP8 MoE op by its explicit overload, so it does not force a host sync per MoE layer. Affects: FP8 per-channel MoE models (for example FP8 Qwen3 MoE, GLM and DeepSeek checkpoints).
- [laguna-moe-sigmoid-routing](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/laguna-moe-sigmoid-routing/README.md): MoE top-k routing ignores `scoring_func`, the score correction bias and `renormalize`. Affects: MoE models with sigmoid scoring, an `e_score_correction_bias`, or `renormalize=False`.
- [fp8-moe-gather](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/fp8-moe-gather/README.md): Backport the gathered-expert FP8 MoE combine from vllm-gaudi v0.30.0 to v0.26.0: at low token counts only the routed experts are read and computed, which speeds up decode on fine-grained FP8 MoE models. Affects: block-quantized FP8 MoE checkpoints with dynamic activation scales and silu experts, for example Qwen3 / Qwen3.5 / Qwen3.6 MoE FP8 (performance).
- [moe-router-weight-on-input](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/moe-router-weight-on-input/README.md): Apply the router weight to the expert input, not the output, for MoE layers that ask for it (`apply_router_weight_on_input`), as Llama 4 does. Affects: Llama 4 Scout and Maverick (BF16 and compressed-tensors checkpoints).
- [mm-merge-idxcopy](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/mm-merge-idxcopy/README.md): Merge multimodal embeddings with `nonzero` + `index_copy_` instead of a broadcast `masked_scatter_` that fails to compile on Gaudi under concurrent image traffic. Affects: vision-language models (images or video).
- [mm-encoder-budget](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/mm-encoder-budget/README.md): Bound the vision-encoder batch with `VLLM_HPU_MM_ENCODER_MAX_PATCHES`, so one request with many large images does not ask for tens of GiB in a single encoder call. Affects: vision-language models with variable-size images (Qwen3-VL and similar).
- [mm-pooling-inputs](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/mm-pooling-inputs/README.md): The pooling (embedding) runner runs the vision encoder and uses M-RoPE positions, so multimodal embedding models such as Qwen3-VL-Embedding embed images instead of crashing or ignoring them. Affects: multimodal pooling models (`--runner pooling`), M-RoPE models in particular.
- [glm4-moe-3d-input](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/glm4-moe-3d-input/README.md): `Glm4MoE.forward` accepts the 3-D hidden states the HPU runner passes, so GLM-4.5 family MoE models generate instead of stopping the engine on the first token. Affects: models served through vLLM's `glm4_moe.py` MoE block: GLM-4.5, GLM-4.5-Air, GLM-4.5V, GLM-4.6.
- [rope-cpu-cache](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/rope-cpu-cache/README.md): Build rotary-embedding cos/sin tables on the CPU, because Gaudi fp32 cos/sin of large arguments are inaccurate. Affects: RoPE models served with long contexts (above about 128K positions).
- [rank-exception-fatal](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/rank-exception-fatal/README.md): Stop the engine when a tensor-parallel worker fails a call whose result is read from another rank, instead of serving with the ranks out of step. Affects: every multi-card deployment that uses vLLM's multiprocessing executor (TP > 1, the default executor on one node).
- [vllm-long-prefill-share](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/vllm-long-prefill-share/README.md): Let long prompts take at most a set share of wall time while other requests decode, so one very long prefill does not stall every other stream. Affects: long-context deployments (hundreds of thousands of tokens) with chunked prefill and concurrent users.
- [pipeline-parallel](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/pipeline-parallel/README.md): Make `--pipeline-parallel-size` greater than 1 work on the HPU runner, with the stages on different nodes linked over the ordinary host network. Affects: anyone who needs a model split by layers across nodes, for example a checkpoint that does not fit one 8-card node.
- [structured-outputs-auto-whitespace](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/structured-outputs-auto-whitespace/README.md): Allow `disable_any_whitespace` with the default `auto` structured-output backend, and make the outlines backend honour it. Affects: servers that want compact JSON from structured outputs (`disable_any_whitespace`).
- [xgrammar-generation-eos-stop](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/xgrammar-generation-eos-stop/README.md): Let a completed grammar end the turn with the model's real end-of-turn tokens, not only the tokenizer's EOS. Affects: structured outputs and forced tool calls (xgrammar backend) on models whose turn ends with a token other than `tokenizer.eos_token_id`.
- [glm47-streaming-text-markers](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/glm47-streaming-text-markers/README.md): Make the `glm47` tool parser recognise tool-call markers written as plain text when streaming, so streaming and non-streaming responses agree. Affects: GLM-4.7 and GLM-5 family models served with `--tool-call-parser glm47`, `stream: true`, forced tool calls.
- [glm45v-box-tokens](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/glm45v-box-tokens/README.md): The `glm45` reasoning parser consumes GLM-4.5V's `<|begin_of_box|>` and `<|end_of_box|>` answer markers instead of passing them to the client. Affects: GLM-4.5V with `--reasoning-parser glm45`.
- [gptoss-harmony-parse-recovery](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/gptoss-harmony-parse-recovery/README.md): Keep a malformed gpt-oss message header from failing the whole request with HTTP 500, and keep the tool call it was meant to carry. Affects: gpt-oss (20B and 120B) with the harmony parser, mostly requests with tools.
- [gemma4-required-tool-grammar](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/gemma4-required-tool-grammar/README.md): Make `tool_choice: "required"` and named tool choice produce a tool call with the `gemma4` tool parser. Affects: Gemma 4 models served with `--tool-call-parser gemma4` and forced tool choice.
- [gemma4-buffered-tool-args](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/gemma4-buffered-tool-args/README.md): Emit `gemma4` tool-call arguments once, complete, with spurious escapes removed. Affects: Gemma 4 models served with `--tool-call-parser gemma4` (code-editing and file-writing tools).
- [gemma4-fast-prefill](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/gemma4-fast-prefill/README.md): Make `--kv-sharing-fast-prefill` work on HPU for Gemma 4 E2B and E4B: during prefill only each prompt's last token runs the KV-shared layers, which cuts prompt compute about 3x on E2B. Affects: Gemma 4 models with KV-shared layers (`num_kv_shared_layers` > 0: E2B, E4B).
- [qwen3-strict-think-end](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/qwen3-strict-think-end/README.md): End Qwen3 reasoning only at `</think>`, so a tool-call example the model quotes while thinking is not returned as a real tool call. Affects: Qwen3-family models served with `--reasoning-parser qwen3` and tool calling, for example Qwen3.6-27B.
- [llama4-tools-lenient](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/llama4-tools-lenient/README.md): A tool-call parser plugin and chat template for Llama 4 that read every call syntax the model writes and render tool history the way the model expects. Affects: Llama 4 Scout and Maverick with tool calling.
- [gemma4-bos-tokenizer](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/fixes/gemma4-bos-tokenizer/README.md): Build a tokenizer directory for Gemma 4 instruct checkpoints that adds `<bos>` to raw prompts, so `/v1/completions` works. Affects: Gemma 4 instruct checkpoints (`google/gemma-4-*-it`) served with `/v1/completions` or `/tokenize`.

## Kernels

- [Kernels](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kernels/index.md): Custom TPC kernels for Gaudi 2.
- [MXFP4 MoE decode kernels](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kernels/mxfp4-moe.md): Up to 3x faster MXFP4 MoE decode on Gaudi 2.

## Ports

- [Ports](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/index.md): Models that released vllm-gaudi images cannot serve, brought up on Intel Gaudi 2.
- [Muse Spark Open on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/muse-spark-open.md): Muse Spark Open is an 816B-parameter mixture-of-experts model (42B active) with a latent MoE in every layer, a norm on each expert's output, and interleaved sliding-window and global attention.
- [GLM-5.3-Flash on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/glm-5-3-flash.md): GLM-5.3-Flash is a 320B-total, 18B-active multimodal MoE model built from four subsystems that rarely appear together: Kimi Delta Attention (KDA, a linear-attention recurrence with per-channel decay), DeepSeek sparse attention (DSA) over an MLA cache with no RoPE, manifold hyper-connections (mHC, four residual streams mixed through a Sinkhorn projection) and a 288-expert MoE.
- [DeepSeek-V4.1-Flash on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/deepseek-v4-1-flash.md): DeepSeek-V4.1-Flash is a 40-layer MoE model (384 routed experts, top-6) with an unusual stack: compressed sparse attention with a learned indexer, an encoder-decoder split in which layers 21 to 39 all read one compressed KV cache written by layer 20, hyper-connections (mHC) normalised by a Sinkhorn loop, and two hashed Engram embedding tables of about 200 GB.
- [GLM-5.2 and GLM-5.3 (744B) in MXFP4 on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/glm-5-mxfp4.md): GLM-5.2 and GLM-5.3 share one architecture: 78 layers, hidden size 6144, MLA attention with DeepSeek sparse attention (DSA, top 2,048 keys), and 256 routed experts plus one shared expert, top-8.
- [Beam-1.0-501B-A23B on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/beam-1-0.md): Beam-1.0 is a 501B-parameter MoE model (23B active) with 52 layers: 3 dense and 49 MoE layers of 256 experts, top-8 with group-limited routing, and grouped-query attention in which 43 layers use a 512-token sliding window with RoPE and 9 use full attention without positional encoding (NoPE) plus a length-dependent query scale.
- [MiMo-V2.6-Flash on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/mimo-v2-6-flash.md): MiMo-V2.6-Flash is a 309B-parameter mixture-of-experts model (15B active) whose attention is mostly a 128-token sliding window with a learned attention sink.
- [Kimi-K2.7-Code on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/kimi-k2-7.md): Kimi-K2.7-Code is a 1T-parameter multimodal MoE model: a DeepSeek-V3-style text tower (61 layers, 384 routed experts, MLA attention) plus a vision tower.
- [MiniMax-M3 with FP8 experts on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/minimax-m3.md): MiniMax-M3 is a 427B-parameter multimodal MoE (23B active): 60 layers, 128 routed experts with top-4 sigmoid routing and a score correction bias, one shared expert, and gpt-oss's SwiGLU-OAI activation (clamp to 7, alpha 1.702).
- [Inkling-Small on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/ports/inkling-small.md): Inkling-Small is a 276B-parameter mixture-of-experts model (12B active) from Thinking Machines.

## Results

- [Results](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/index.md): Studies, features and findings from running a large Gaudi 2 fleet that are not tied to a single model port.
- [What Gaudi 2 gets wrong, and the workaround](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/platform-findings.md): Defects found below vLLM while porting and serving models on Gaudi 2, in the Habana PyTorch bridge, the FusedSDPA kernel, the runtime and the driver.
- [Prefix caching on Gated DeltaNet hybrids](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/gdn-prefix-caching.md): The Qwen3.5, Qwen3.6 and Qwen3.8 families mix Gated DeltaNet (GDN) linear-attention layers with full attention.
- [Anatomy of a large-MoE decode step](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/moe-decode-step.md): Where does one decode step of a large expert-parallel MoE go on 8 Gaudi 2 cards, and which parts can be moved? This page is a method for answering that on any EP MoE, worked through on GLM-5.3 (MXFP4 experts, TP8 + EP8, 256 routed experts, top-8, MTP with one draft token).
- [Two-node pipeline parallel (PP2 x TP8)](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/two-node-pipeline.md): A 744B GLM-5.3 does not fit one 8-card Gaudi 2 node in FP8, which is why the single-node build serves MXFP4 experts (GLM-5.2 and GLM-5.3 in MXFP4).
- [Gemma-4-E2B fast prefill on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/gemma-4-e2b-prefill.md): Gemma-4-E2B has 35 decoder layers, and the last 20 write no KV cache of their own: they read the cache of an earlier layer.
- [Fast text-to-speech on Gaudi 2](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/tts-on-gaudi.md): Three open text-to-speech models stream faster than real time from one Gaudi 2 card.
- [EmbeddingGemma 2 on Gaudi 2 without vLLM](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/results/embeddinggemma-2.md): EmbeddingGemma 2 is a 740M multimodal embedding model: text, images, audio and video, alone or mixed, into one 768-dimension vector.

## Kubernetes

- [Kubernetes](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/index.md): This section covers running vLLM on Gaudi at fleet scale on Kubernetes: what a Gaudi pod needs, how to place pods so cards do not fragment, how to start, drain, roll and autoscale model servers that take minutes to load, and how to notice when one stops working while still looking healthy.
- [Pod spec](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/pod-spec.md): What a vLLM pod on Gaudi needs, and why.
- [Scheduling](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/scheduling.md): How to place Gaudi pods so the fleet does not fragment, and why model pods should never be pinned to nodes.
- [Start-up and warm-up](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/startup-and-warmup.md): A model server's cold start decides how fast the fleet can react: how quickly an autoscaler can follow a burst, how long a rollout takes, and how much a node failure costs.
- [Health and failure](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/health-and-failure.md): The failures that cost the most are the ones every layer reports as healthy.
- [Rollouts and scaling](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/rollouts-and-scaling.md): Model servers take minutes to start and hold long streams open.
- [Routing](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/routing.md): How requests reach model pods matters as much as how many pods there are.
- [Observability](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/observability.md): What to scrape, what to alert on, and how to keep logs and Prometheus useful on a large fleet.
- [Multi-node replicas](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/docs/kubernetes/multi-node.md): How to run one model replica across two 8-card nodes, and what changes on Kubernetes when a replica is a group of pods instead of one pod.

## Optional

- [Kubernetes manifests and tools](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/kubernetes/README.md): what each manifest and tool does; placeholders to replace.
- [MXFP4 kernel sources and build](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/kernels/mxfp4-moe/README.md): layout, toolchain, build, enable, verify.
- [Servers](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/servers/README.md): models that do not run on vLLM (text-to-speech, EmbeddingGemma 2).
- [Contributing](https://raw.githubusercontent.com/thediymaker/gaudi-cookbook/main/CONTRIBUTING.md): ground rules for changes.