Ornith-1.0-35B¶

Ornith's 35B hybrid MoE agentic coding model (post-trained from Qwen 3.5: 30 Gated DeltaNet and 10 attention layers, 256 experts, 3B active) in BF16 with its vision tower, at its native 256K context on two Gaudi 2 cards, 16 sequences at a time. Prefix caching is off on this release (see Prefix caching).


Model	ornith-ai/Ornith-1.0-35B (MIT)
Hardware	2x Gaudi 2 (TP 2)
Image	vllm-gaudi `v0.26.0` (`b059f9c`), vLLM `568afb3a`, Gaudi SW 1.24.1
Precision	BF16 (70 GB of weights including the vision tower)
Context	262,144 (native)
Concurrency	`--max-num-seqs 16`
Parsers	tools `qwen3_xml`, reasoning `qwen3`
Recipe	`recipes/ornith-ai/Ornith-1.0-35B/recipe.yaml`

Prefix caching¶

Prefix caching is off: on this release, prefix-cache hits on Gated DeltaNet layers resume from the wrong state. Upstream vllm-gaudi main has merged prefix caching for the non-compact layout; use a release that includes it before turning prefix caching on.

Launch¶

$COOKBOOK is a checkout of this repository inside the container.

export PT_HPU_LAZY_MODE=0
export PT_HPU_ENABLE_LAZY_COLLECTIVES=true
export VLLM_USE_NAIVE_MAMBA_CACHE_SHARING=1
export VLLM_COMPACT_GDN=1
export VLLM_SKIP_WARMUP=true
export VLLM_ENGINE_ITERATION_TIMEOUT_S=7200
export VLLM_RPC_TIMEOUT=7200000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.26.0 profile, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.26.0 || exit 1

vllm serve ornith-ai/Ornith-1.0-35B \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.9 \
  --max-num-batched-tokens 1024 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --limit-mm-per-prompt '{"image": 16}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --trust-remote-code

Compile mode (PT_HPU_LAZY_MODE=0), with the compact GDN state layout that is the stock default without prefix caching, as for the other Qwen3.5-family recipes. On an earlier vllm-gaudi release lazy mode failed the first decode at TP 2 in the GDN layers (tensor_data is empty in ValidateSyncInputTensors).
No VLLM_*_BUCKET_* variables: compile mode, default buckets.
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800: with warm-up skipped every new shape compiles inside the model call, and the 300 s default can stop the engine mid-compile.
The full model is served, vision tower included (no --language-model-only), with up to 16 images per prompt: the worst-case context clamp at 16,384 tokens per maximum-size image.
qwen3_xml is the tool parser for the model's <tool_call> XML format.

Buckets¶

No bucket file and no bucket variables.

Required fixes¶

All from Fixes, all in --profile v0.26.0, which also applies the release's other generic fixes (see Using a recipe). The profile also runs seeded-sampling, prompt-order and resumed-prefill-logits; this recipe does not depend on them (the seed segfault is a lazy-mode bug, and prefix caching is off).

Fix	Why this model needs it
`sampling-rows-refresh`	stale device sampling rows after a batch reorder
`structured-output-rows`	structured-output steps sample each row with another request's parameters
`bad-words-index`	`bad_words` raises `IndexError` in the engine core
`logitsprocs-rows`	`logit_bias`, `min_p` and `min_tokens` are silently ignored
`async-discard-counter`	`AttributeError` on the first preemption under async scheduling
`hybrid-decode-pad`	when the hybrid KV block is split into 128-token kernel blocks, decode padding writes into a live block of another request
`gather-logprobs-int32`	the id corruption is lazy-mode only, but the fix is harmless here and keeps one fix set across recipes
`prompt-logprobs`	multimodal model with one prompt in a prefill batch returns flat hidden states; the first `prompt_logprobs` request kills the engine
`logprobs-decode-guard`	a bad logprob id from any source is replaced instead of stopping the API server
`rank-exception-fatal`	an exception on a non-zero TP rank stops the engine instead of serving from out-of-step ranks
`rope-cpu-cache`	Gaudi fp32 cos/sin are wrong past about 1.5e5, so RoPE tables built on the HPU are wrong from position about 196K

Verified¶

On vllm-gaudi v0.26.0, two Gaudi 2 cards, with exactly the launch command above in a container started from the stock image (the fixes applied at start-up). Requests went to the OpenAI-compatible API.

Check	Result
Known-answer prompts (arithmetic, facts, translation), temperature 0	8/8
The same prompts, 16 in flight	16/16
Tool call, `tool_choice: auto`, sampled at 0.6	5/5 valid `get_weather` calls with the right argument
Streamed tool call	name and JSON arguments assembled correctly
Tool result fed back	the answer uses the returned values
`top_logprobs: 20`	20 entries on every token
`prompt_logprobs: 5`	returned; server healthy afterwards
`response_format` JSON schema	valid object
`logit_bias` +100 on one token	takes effect
4 images, one solid colour each	colours named in order
16 images of 448 x 448 in one prompt	16/16 colours named in order
Needle in a haystack, 3 depths, cold	3/3 at 65K, 131K and 244K tokens
Server health after all checks	healthy, no restarts

KV pool at start-up: 5,162,307 tokens.

Performance¶

Single runs on the server above (measured as: per-call, one server; times are end to end and include time to first token).

Test	Result
One stream, short prompt, 256 tokens	23.2 tokens/s
16 streams, short prompts, 256 tokens each	298.2 tokens/s aggregate, 18.6 per stream
Cold prompt of 65K tokens (needle)	15 to 17 s (65 s for the first, which also compiled its graphs) to the answer
Cold prompt of 131K tokens (needle)	32 to 45 s to the answer
Cold prompt of 244K tokens (needle)	73 to 85 s to the answer

Known issues¶

Give test requests a real token budget: the model reasons before answering.

What not to do¶

Do not use lazy mode.
Do not turn on prefix caching on this release.