openai/gpt-oss-120b

117B MoE (5.1B active) reasoning model in native MXFP4, harmony format, 128K context on four Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide

moe117B / 5.1B active131,072 ctx servedmxfp4vllm-gaudi v0.26.0Text

Hardware

4 cards · TP 4 · EP

Image

vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1

Variant

The original MXFP4 MoE checkpoint (65 GB), kept packed on the device; four Gaudi 2 cards with expert parallelism.

Features

Fixes

profile v0.26.0 gptoss-harmony-parse-recovery

15 fixes this configuration depends on; opt-in fixes are named.

Launch

Shell

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "openai/gpt-oss-120b", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "max_tokens": 512}'

export PT_HPU_LAZY_MODE=1
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export HPU_MXFP4_NATIVE=1
export HPU_FUSED_MOE=1
export PT_HPU_QKV_SLICE_SEQ_LEN_THLD=128
export TIKTOKEN_RS_CACHE_DIR=/data/tiktoken-rs-cache
export VLLM_SKIP_WARMUP=true
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=16
export VLLM_DECODE_BS_BUCKET_MIN=1
export VLLM_DECODE_BS_BUCKET_STEP=1
export VLLM_DECODE_BS_BUCKET_MAX=16
export VLLM_ENGINE_ITERATION_TIMEOUT_S=7200
export VLLM_RPC_TIMEOUT=7200000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

mkdir -p "$TIKTOKEN_RS_CACHE_DIR"

# Fixes: the v0.26.0 profile plus this recipe's opt-in fixes, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 gptoss-harmony-parse-recovery || exit 1

vllm serve openai/gpt-oss-120b \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --max-model-len 131072 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.9 \
  --block-size 128 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice \
  --tool-call-parser openai \
  --reasoning-parser openai_gptoss \
  --trust-remote-code

Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.

At a glance

Checks that passed

16 in flight, about 4.1K-token prompts and 1.7K-token outputs: 96/96; two about 107K-token prompts 2/2 (TTFT 78-99 s)
Request options: structured output, named and required tool calls through harmony, min_tokens, min_p, logit_bias all take effect
Exact sinks and sliding window: long-context needles 9/9 with sink-once and swa-decode-window, 5/9 without
Model test suite with the sink fixes 17/17 twice (14/17 and 15/17 without); coding-agent tier 5/5 twice (2/5 and 3/5 without)
Gates at 32 seqs: harmony tools 8/8 per case, json_schema 6/6, two about 112K-token needles found, GSM8K 49/50

Notes

openai_harmony downloads its vocab at start and does not create its cache directory: mkdir -p "$TIKTOKEN_RS_CACHE_DIR" first, on storage that survives restarts.
HPU_MXFP4_NATIVE=0 dequantizes to BF16 and lands on a fused MoE kernel larger than the Gaudi 2 TPC instruction cache (device fault).
The cookbook MXFP4 MoE decode kernel (kernels/mxfp4-moe) is 2-3x faster per stream at 4-16 in flight and makes two cards practical; this recipe is the stock path.

Guide

OpenAI's 117B mixture-of-experts reasoning model (5.1B active, harmony format), served from its native MXFP4 weights at 128K context on four Gaudi 2 cards with expert parallelism. This is the stock path; the cookbook's MXFP4 MoE kernel makes it faster (see the end of the page).


Model	openai/gpt-oss-120b
Hardware	4x Gaudi 2 (TP 4, `--enable-expert-parallel`)
Image	vllm-gaudi `v0.26.0` (`b059f9c`), vLLM `568afb3a`, Gaudi SW 1.24.1
Precision	MXFP4 MoE weights kept packed on the device (65 GB checkpoint)
Context	131,072 (native)
Concurrency	`--max-num-seqs 16`
Parsers	tools `openai`, reasoning `openai_gptoss`
Recipe	`recipes/openai/gpt-oss-120b/recipe.yaml`

Required fixes

All from Fixes. The ones marked opt-in are named in the launch command; the rest come from --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe).

Fix	Why this model needs it
`prompt-order`	prefix caching can read another request's KV in the same step
`sampling-rows-refresh`	stale device sampling rows after a batch reorder
`structured-output-rows`	structured-output steps sample each row with another request's parameters
`bad-words-index`	`bad_words` raises `IndexError` in the engine core
`logitsprocs-rows`	`logit_bias`, `min_p` and `min_tokens` are silently ignored
`async-discard-counter`	`AttributeError` on the first preemption under async scheduling
`resumed-prefill-logits`	a request resumed after preemption with prefix-cache hits samples again at every recomputed position, so the generated tail repeats
`gather-logprobs-int32`	in lazy mode the `top_logprobs` ids come back corrupted
`prompt-logprobs`	the first `prompt_logprobs` request kills the engine
`logprobs-decode-guard`	a bad logprob id is replaced instead of stopping the API server
`seeded-sampling`	a request with `seed` can segfault a TP worker in lazy mode
`sink-once`	attention sinks: the prefill kernel ignores the sink argument, and paged decode adds the sink once per KV block (about 80 copies at 10K context)
`swa-decode-window`	the 128-token sliding layers saw 129 to 256 tokens in decode
`rank-exception-fatal`	an exception on a non-zero TP rank stops the engine instead of leaving ranks out of step
`gptoss-harmony-parse-recovery` (opt-in)	one malformed tool-call header raised out of the harmony parser and turned the whole request into HTTP 500 (5/5 with a system prompt and 3 tools)

logits-empty-guard is in the profile but not needed by this recipe: with async scheduling (the default) a prompt chunk that does not finish its prompt keeps one logits row, so the tensor-parallel gather is never empty. It becomes necessary if you add --no-async-scheduling.

Verified

On vllm-gaudi v0.26.0, four Gaudi 2 cards, TP 4 with expert parallelism:

16 in flight with about 4.1K-token prompts and 1.7K-token outputs: 96/96. Two prompts of about 107K tokens: 2/2 (TTFT 78 to 99 s).
Request options: structured output, named and required tool calls through harmony, min_tokens, min_p and logit_bias all take effect.
With sink-once and swa-decode-window: long-context needles 9/9 (5/9 without), the model test suite 17/17 on two runs (14/17 and 15/17 without), coding-agent tier 5/5 twice (2/5 and 3/5 without). The failures without them were malformed tool JSON and leaked headers.
At --max-num-seqs 32: harmony tools 8/8 per case (one-shot, required, tool-result turn, second-call turn), json_schema 6/6, two needles at about 112K tokens found, GSM8K 49/50.

Performance

Single measurements on four cards, vllm-gaudi v0.26.0, stock MoE op, about 3.2K-token prompts and 1,024 output tokens (run at --max-num-seqs 32).

In flight	Tokens/s per stream	Tokens/s per replica	TTFT
1	57.0	57	0.61 s
4	20.1	78	1.94 s
16	10.0	150	6.57 s
32	8.9	253	13.20 s

At --max-num-seqs 16 with production-shaped requests (about 4.1K-token prompts, 1.7K-token outputs) a replica gave 71.7 tokens/s at 4 in flight and 133.9 at 16. The KV pool holds 3.7M tokens, so concurrency is limited by decode speed, not memory.

Faster: the MXFP4 MoE kernel

The MoE is most of a decode step. The cookbook's MXFP4 MoE decode kernel replaces the stock op for decode steps of up to 32 tokens and reads the packed weights directly. Measured on the same server (TP 4, expert parallel), per stream: 62.6 vs 57.0 at 1 in flight, 59.6 vs 20.1 at 4, 22.4 vs 10.0 at 16. With it, two cards (TP 2) give 52 tokens/s per stream at 4 in flight, where this four-card stock recipe gives 20. Prefill is unchanged.

Known issues

Multi-turn agent sessions in harmony still have rough edges with some clients; keep both parsers on and the harmony recovery fix in place.
Penalties are not applied under async scheduling on v0.26.0 (see Qwen3-30B-A3B-2507).

What not to do

Do not run the tool parser without --reasoning-parser openai_gptoss: the analysis channel is then never split off and its text lands inside the tool arguments. A short single call still looks fine, so simple tests miss it.
Do not set HPU_MXFP4_NATIVE=0 on Gaudi 2. The weights are then dequantized to BF16 and the MoE lands on a fused BF16 TPC kernel larger than the 64 KB instruction cache, which faults the device. The old workaround (a non-fused per-expert Python loop) costs about 3x throughput.
Do not leave the harmony vocabulary cache in the container's /tmp: every start then downloads it again, and a truncated download is fatal before the server binds.