Skip to content

Gaudi Cookbook

Practical, tested ways to serve open LLMs with vLLM on Intel Gaudi 2, through the vllm-gaudi plugin.

Everything here was run on a large Gaudi 2 fleet serving real traffic, and everything uses public images and public checkpoints.

New: MXFP4 MoE decode kernels

Up to 2.5x faster decode for MXFP4 mixture-of-experts models on one 8-card Gaudi 2 node: GLM-5.3 single stream from 23.3 to 59.1 tokens/s, Kimi-K2.7 from 17.9 to 44.5, Muse Spark Open 2.4x aggregate at four streams, with lower numerical error than the stock op. Read the write-up.

  • Recipes

    Copy-paste vllm serve commands per model: cards, precision, buckets, parsers, the checks that passed and the pitfalls to avoid.

  • Fixes

    Small runtime fixes for bugs in released vllm-gaudi images, each linked to its upstream pull request. They refuse to start if the image does not match.

  • Kernels

    MXFP4 MoE decode kernels that read the packed weights directly: up to 2.5x faster decode on GLM-5.x, Kimi-K2.7, Muse Spark Open and DeepSeek-V4.1-Flash.

  • Ports

    Seven models that released images cannot serve, brought up on Gaudi 2: Muse Spark Open, GLM-5.3-Flash, DeepSeek-V4.1-Flash, Beam-1.0 and more.

  • Results

    Platform findings, prefix caching on hybrids, MoE decode-step anatomy, two-node pipeline parallel, fast prefill, speech and embeddings.

  • Kubernetes

    Running Gaudi at fleet scale: pod spec, bin-pack scheduling, warm-up, health, rollouts, autoscaling and routing.

  • Use with an agent

    Point Claude Code, Codex, Cursor or another coding agent at this repository. AGENTS.md tells it where everything is and what it must not do.

Start here

  1. Check the software baseline and build or pull a matching image.
  2. Pick a recipe and apply its required fixes.
  3. Moving to Kubernetes? Read the pod spec page before anything else.
  4. Working with a coding agent? Point it at this repository; see Use with an agent.

Status

This is an independent project. It is not affiliated with Intel or the vLLM project. Fixes and kernels here are offered upstream first; when a change ships in a vllm-gaudi release, the matching fix is marked as superseded.