meta-models/Muse-Glimmer-30B

Checks that passed

• Port image path (install.sh on the stock image, then the v0.30.0 profile plus topk-subset-sampler): 11/11 device gates in tests/hpu/run_gates.sh
• 32 concurrent requests 32/32 correct; stock v0.30.0 stops the engine on the first two concurrent requests
• Needles at 8K, 32K (5% and 95% depth), 120K and 127K prompt tokens: 5/5
• Vision: shape count and OCR of a rendered image, 8 image sizes 8/8, text-only request after images correct
• tool_choice required 24/24 (8 greedy, 16 sampled at the generation defaults), named call with schema-valid arguments, auto still answers in prose; streaming required call has id, name and arguments
• logprobs 5 on completions 3/3 clean with greedy top-1 at every position; chat top_logprobs; seeded requests 128/128 at 64 concurrent, same-seed repeats identical
• KV cache 1,571,584 tokens (11.99 sequences of 131,072)

Notes

• The port adds three model-only patches to the stock image: the attention output keeps its 3D shape (batch > 1 stops the engine without it), the sliding layers read only their window in chunked prefill (without it a 120K prompt returned garbage and corrupted later requests), and tool_choice required/named are enforced with a grammar.
• Sampling defaults from generation_config.json are temperature 1.0, top_p 0.95, top_k 64.
• response_format json_schema is not enforced on this model: the grammar waits for a reasoning end that a plain answer never sends.
• tool_choice none can leak ATEM tool markup into content when the model calls a tool anyway.

Meta's 29.6B dense vision-language agent model (52 layers, 3:1 sliding-window and global attention, a 50-layer vision encoder), served in BF16 at 128K context on two Gaudi 2 cards with up to eight images per prompt. vLLM ced6857a already has the model and both parsers; a small port adds three model-only patches that the stock releases/v0.30.0 image needs on Gaudi.

Build the image

From a checkout of this repository, on top of a stock releases/v0.30.0 image built as in Getting started:

docker build -f ports/muse-glimmer-30b/Dockerfile --build-arg BASE=<your-vllm-gaudi-v0.30.0-image> -t <your-muse-glimmer-30b-image> .

install.sh checks the vllm-gaudi and vLLM commits and applies three patches, each fail-closed with --check:

• Attention output rank. The model multiplies its sigmoid output gate, computed from the 3D [batch, query, hidden] layer input, by the HPU attention output, which comes back flattened as [batch * query, hidden]. Batch 1 broadcasts by accident, so single requests work and the first two concurrent requests stop the engine (The size of tensor a (128) must match the size of tensor b (256)). The patch restores the rank once, after the attention call.
• Sliding-window blocks in chunked prefill. vllm-gaudi builds the window block list for sliding layers only when model_type is gemma4; the patch adds muse_glimmer. Without it the 39 sliding layers read the whole context in every 8K prefill chunk. On the same image without the patch, a 120K-token prompt returned one word repeated to max_tokens, every later request on that server returned garbage, and the engine then failed with PT_DEVMEM Allocation failed (reproduced twice; not root-caused further).
• Forced tool calls. The muse_glimmer tool parser skips vLLM's JSON-schema path for tool_choice "required" and named functions (the model writes ATEM XML) and installs nothing instead, so those choices decode like "auto". The patch installs an xgrammar structural tag in the model's own channel format, applied from the first token.

The image carries this repository at /opt/gaudi-cookbook. Details: the port README.

Required fixes

The v0.30.0 profile plus one opt-in fix. The ones this configuration depends on:

Client usage

• The model card's sampling is temperature 1.0, top_p 0.95, top_k 64; these are the server defaults.
• The model reasons in a separate channel before answering, even for one-word answers. Give it room: a reply cut off in the reasoning channel has content: null.
• tool_choice "required" and named functions are enforced; the model may still reason first, which costs about 150 to 300 tokens per forced call.

Verified

On the port image path exactly as written (stock releases/v0.30.0 image, install.sh, the fix step, the serve command), two Gaudi 2 cards, ports/muse-glimmer-30b/tests/hpu/run_gates.sh: 11/11 PASS.

Performance

One server, two cards, the Launch command above, synthetic chat prompts of about 285 tokens, 512 output tokens per request (ignore_eos), sampling at the generation-config defaults, measured after a warm-up round. Per stream is output tokens over each request's wall time, first token included. Single runs.

Known issues

• response_format with a JSON schema is not enforced: the grammar waits for the reasoning parser's end signal, which a plain answer never sends. Asked for {name, year: integer}, the model returned a fenced block with other keys and "753 BC" as a string.
• tool_choice: "none" keeps the tool schemas in the prompt and runs no parser, so a model that calls a tool anyway returns its raw ATEM markup in content.
• DFlash speculative decoding (the model's draft head in vLLM) was not tried on Gaudi.

What not to do

• Do not drop the port and serve the stock image: one concurrent request pair stops the engine.
• Do not lower --api-server-count to 1 or drop topk-subset-sampler at high concurrency. On a vLLM 0.26 build of this model each one capped a pod's aggregate throughput (the single frontend process, and the full-vocabulary sort at the default top_k); neither was re-measured on this image.
• Do not enable only one of the two muse_glimmer parsers.