Gaudi Cookbook / Recipes
/

Qwen/Qwen3-VL-Embedding-8B

8B multimodal embedding model (text, images, screenshots; 4096 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.

Verified on Gaudi 2 Hugging Face Full guide
dense8B32,768 ctx servedbf16vllm-gaudi v0.30.0EmbeddingMultimodal
Hardware
1 card · TP 1
Image
vllm-gaudi releases/v0.30.0 @ 058f107 · vLLM @ ced6857a · Gaudi SW 1.24.1
Variant
BF16 weights (16.3 GB), one Gaudi 2 card.
Fixes
profile v0.30.0mm-pooling-inputs
2 fixes this configuration depends on; opt-in fixes are named.
Launch
Shell 
export PT_HPU_LAZY_MODE=1
export PT_HPUGRAPH_DISABLE_TENSOR_CACHE=false
export VLLM_SKIP_WARMUP=true
export VLLM_MULTIMODAL_BUCKETS=None
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

# Fixes: the v0.30.0 profile plus this recipe's opt-in fixes, in the pack's order.
# Stops with a non-zero exit code if the image does not match.
python3 "$COOKBOOK/fixes/apply_all.py" --profile v0.30.0 mm-pooling-inputs || exit 1

vllm serve Qwen/Qwen3-VL-Embedding-8B \
  --tensor-parallel-size 1 \
  --runner pooling \
  --max-model-len 32768 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.80 \
  --no-enable-prefix-caching \
  --no-enable-chunked-prefill \
  --chat-template "$COOKBOOK/recipes/Qwen/Qwen3-VL-Embedding-8B/embed.jinja" \
  --limit-mm-per-prompt '{"image":32,"video":0}' \
  --hf-overrides '{"is_matryoshka": true, "matryoshka_dimensions": [64, 128, 256, 512, 1024, 1536, 2048, 3072, 4096]}'
Run inside a vllm-gaudi container. $COOKBOOK is a checkout of this repository there.
At a glance

Checks that passed

  • Stock image without mm-pooling-inputs: EngineDeadError on the first request
  • Served vectors vs a CPU FP32 transformers reference: text 0.9961 / 0.9935, image 0.9885 / 0.9915 cosine
  • COCO val2017 39769 vs captions: matching 0.69, unrelated 0.13; four different images mean pairwise cosine 0.169
  • Flickr30k 1k test, first 300 images: caption to image R@1 81.5, image to caption R@1 96.0
  • SciFact nDCG@10 with the instruction as the system message: 78.9

Notes

  • Send inputs as chat messages: system = the instruction (queries) or "Represent the user's input." (documents), user = text and/or images.
  • Plain-string input gets no chat wrapping and a different last token; do not mix it with messages in one index.
Guide

Qwen's 8B multimodal embedding model (text, images and screenshots in one 4,096-dimension space, Matryoshka truncation), served through vLLM's pooling runner in BF16 on a single Gaudi 2 card. It needs one fix that is not in the release profile, mm-pooling-inputs, and a chat template that ends every prompt with the token the model card pools.
Model Qwen/Qwen3-VL-Embedding-8B
Hardware 1x Gaudi 2 (TP 1)
Image vllm-gaudi releases/v0.30.0 (058f107), vLLM ced6857a, Gaudi SW 1.24.1
Precision BF16 (16.3 GB of weights)
Context 32,768 tokens per input (262,144 native)
Concurrency --max-num-seqs 8
Endpoint /v1/embeddings with messages
Recipe recipes/Qwen/Qwen3-VL-Embedding-8B/recipe.yaml

Required fixes

All from Fixes. --profile v0.30.0 applies the release's generic fixes; the opt-in fix is named in the launch command.

Fix Why this model needs it
mm-merge-idxcopy image embeddings are merged into the prompt by row index; the broadcast masked_scatter_ fails to compile under concurrent images
mm-pooling-inputs (opt-in) stock v0.30.0 stops the engine on the first request (AttributeError: 'Qwen3VLForEmbedding' object has no attribute 'model'); past that, the pooling path never runs the vision encoder (image placeholders are embedded as text) and uses 1D positions instead of M-RoPE

Client usage

Send inputs as chat messages, as the model card does. The system message is the task instruction for queries and Represent the user's input. for documents; the user message holds text, images, or both.

{"model": "Qwen/Qwen3-VL-Embedding-8B",
 "messages": [
   {"role": "system", "content": [{"type": "text", "text": "Retrieve images that match the user's query."}]},
   {"role": "user", "content": [{"type": "text", "text": "two cats sleeping on a pink couch"}]}]}

An image document is a user message with an image_url part. Add "dimensions": 256 (or another listed size) for a truncated, renormalized vector.

Verified

On vllm-gaudi releases/v0.30.0, one Gaudi 2 card, the launch above.

Check Result
Stock image without mm-pooling-inputs engine stopped on the first request (AttributeError, then EngineDeadError)
Served vectors vs a CPU transformers reference (FP32, last token of the model card's prompt) text 0.9961 and 0.9935; image 0.9885 (COCO photo) and 0.9915 (image plus text) cosine
COCO val2017 39769 (two cats on a pink couch) vs captions matching caption 0.69, unrelated caption 0.13
Four different images mean pairwise cosine 0.169 (the broken path collapses every image onto nearly one vector)
Flickr30k 1k test, first 300 images (1,500 captions) caption to image R@1 81.5, image to caption R@1 96.0
SciFact nDCG@10 (instruction as system message) 78.9
dimensions: 256 256-value vector

Performance

Single measurements on one card, releases/v0.30.0, warm, one request per input.

Load Result
Text (about 30 tokens), 1 request 82 ms p50
Text, 64 requests at 8 concurrent 4.8 s (76 ms per input)
Distinct 640x480 images, 1 request 132 ms p50
Distinct 640x480 images, 64 requests at 8 concurrent 7.2 s (113 ms per input)
SciFact corpus, 5,183 documents at 8 concurrent 447 s (86 ms per document)

Pooling prefill runs one sequence at a time on this image (see Qwen3-Embedding-8B), so concurrency does not raise throughput much. Scale out with replicas.

Known issues

  • A plain-string input gets no chat wrapping, and it is tokenized with one more special token than the same text written out in the template, so its vector differs slightly (cosine 0.994 in one check) from the messages form. Pick one form per index; the model card uses messages.
  • Video inputs are not enabled ("video":0) and were not tested.

What not to do

  • Do not serve without embed.jinja and rely on clients to send add_generation_prompt and add_special_tokens: a client that forgets gets vectors pooled on the wrong token, with no error.
  • Do not mix vectors from this model with Qwen3-Embedding-8B in one index; the spaces differ.