Gaudi Cookbook / Recipes
/

ornith-ai/Ornith-1.0-35B

35B hybrid MoE (Gated DeltaNet + attention, 3B active) agentic coding model with vision, BF16, native 256K context on two Gaudi 2 cards.

Verified on Gaudi 2 Hugging Face Full guide
moe35B / 3B active262,144 ctx servedbf16vllm-gaudi v0.26.0Multimodal
Hardware
2 cards · TP 2
Image
vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1
Variant
BF16 weights with the vision tower (70 GB), two Gaudi 2 cards.
Features
Fixes
profile v0.26.0
11 fixes this configuration depends on; opt-in fixes are named.
Launch
Shell 
export PT_HPU_LAZY_MODE=0
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_USE_NAIVE_MAMBA_CACHE_SHARING=1
export VLLM_COMPACT_GDN=1
export VLLM_SKIP_WARMUP=true
export VLLM_ENGINE_ITERATION_TIMEOUT_S=7200
export VLLM_RPC_TIMEOUT=7200000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 || exit 1

vllm serve ornith-ai/Ornith-1.0-35B \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.9 \
  --max-num-batched-tokens 1024 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --limit-mm-per-prompt '{"image": 16}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --trust-remote-code
Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.
At a glance

Checks that passed

  • Known-answer prompts 8/8 at temperature 0 and 16/16 with 16 in flight
  • Tool calls: 5/5 valid calls sampled at 0.6, streamed call assembled, tool result used in the answer
  • top_logprobs 20, prompt_logprobs, JSON schema and logit_bias all take effect; server healthy after each
  • Vision: 16 images in one prompt, 16/16 colours named in order
  • Needles, 3 depths each, cold: 3/3 at 65K, 131K and 244K tokens
  • Single stream 23.2 tokens/s, 16 streams 298.2 tokens/s aggregate (256 tokens, end to end)

Notes

  • Prefix caching is off, as verified. On the stock release prefix-cache hits on Gated DeltaNet layers resume from the wrong state; gdn-prefix-cache makes them correct on other Qwen3.5-family recipes but has not been run on this model.
Guide

Ornith's 35B hybrid MoE agentic coding model (post-trained from Qwen 3.5: 30 Gated DeltaNet and 10 attention layers, 256 experts, 3B active) in BF16 with its vision tower, at its native 256K context on two Gaudi 2 cards, 16 sequences at a time. Prefix caching is off on this release (see Prefix caching).
Model ornith-ai/Ornith-1.0-35B (MIT)
Hardware 2x Gaudi 2 (TP 2)
Image vllm-gaudi v0.26.0 (b059f9c), vLLM 568afb3a, Gaudi SW 1.24.1
Precision BF16 (70 GB of weights including the vision tower)
Context 262,144 (native)
Concurrency --max-num-seqs 16
Parsers tools qwen3_xml, reasoning qwen3
Recipe recipes/ornith-ai/Ornith-1.0-35B/recipe.yaml

Prefix caching

Prefix caching is off: on this release, prefix-cache hits on Gated DeltaNet layers resume from the wrong state. Upstream vllm-gaudi main has merged prefix caching for the non-compact layout; use a release that includes it before turning prefix caching on.

Required fixes

All from Fixes, all in --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe). The profile also runs seeded-sampling, prompt-order and resumed-prefill-logits; this recipe does not depend on them (the seed segfault is a lazy-mode bug, and prefix caching is off).

Fix Why this model needs it
sampling-rows-refresh stale device sampling rows after a batch reorder
structured-output-rows structured-output steps sample each row with another request's parameters
bad-words-index bad_words raises IndexError in the engine core
logitsprocs-rows logit_bias, min_p and min_tokens are silently ignored
async-discard-counter AttributeError on the first preemption under async scheduling
hybrid-decode-pad when the hybrid KV block is split into 128-token kernel blocks, decode padding writes into a live block of another request
gather-logprobs-int32 the id corruption is lazy-mode only, but the fix is harmless here and keeps one fix set across recipes
prompt-logprobs multimodal model with one prompt in a prefill batch returns flat hidden states; the first prompt_logprobs request kills the engine
logprobs-decode-guard a bad logprob id from any source is replaced instead of stopping the API server
rank-exception-fatal an exception on a non-zero TP rank stops the engine instead of serving from out-of-step ranks
rope-cpu-cache Gaudi fp32 cos/sin are wrong past about 1.5e5, so RoPE tables built on the HPU are wrong from position about 196K

Verified

On vllm-gaudi v0.26.0, two Gaudi 2 cards, with exactly the launch command above in a container started from the stock image (the fixes applied at start-up). Requests went to the OpenAI-compatible API.

Check Result
Known-answer prompts (arithmetic, facts, translation), temperature 0 8/8
The same prompts, 16 in flight 16/16
Tool call, tool_choice: auto, sampled at 0.6 5/5 valid get_weather calls with the right argument
Streamed tool call name and JSON arguments assembled correctly
Tool result fed back the answer uses the returned values
top_logprobs: 20 20 entries on every token
prompt_logprobs: 5 returned; server healthy afterwards
response_format JSON schema valid object
logit_bias +100 on one token takes effect
4 images, one solid colour each colours named in order
16 images of 448 x 448 in one prompt 16/16 colours named in order
Needle in a haystack, 3 depths, cold 3/3 at 65K, 131K and 244K tokens
Server health after all checks healthy, no restarts

KV pool at start-up: 5,162,307 tokens.

Performance

Single runs on the server above (measured as: per-call, one server; times are end to end and include time to first token).

Test Result
One stream, short prompt, 256 tokens 23.2 tokens/s
16 streams, short prompts, 256 tokens each 298.2 tokens/s aggregate, 18.6 per stream
Cold prompt of 65K tokens (needle) 15 to 17 s (65 s for the first, which also compiled its graphs) to the answer
Cold prompt of 131K tokens (needle) 32 to 45 s to the answer
Cold prompt of 244K tokens (needle) 73 to 85 s to the answer

Known issues

  • Give test requests a real token budget: the model reasons before answering.

What not to do

  • Do not use lazy mode.
  • Do not turn on prefix caching on this release.