Gaudi Cookbook / Recipes
/

Qwen/Qwen3-Embedding-4B

4B instruction-aware text embedding model (2560 dimensions, Matryoshka truncation) on one Gaudi 2 card in BF16.

Verified on Gaudi 2 Hugging Face Full guide
dense4B32,768 ctx servedbf16vllm-gaudi v0.26.0Embedding
Hardware
1 card · TP 1
Image
vllm-gaudi v0.26.0 @ b059f9c · vLLM @ 568afb3a · Gaudi SW 1.24.1
Variant
BF16 weights (8 GB), one Gaudi 2 card.
Fixes
None: this recipe runs on the stock image.
Launch
Shell 
export PT_HPU_LAZY_MODE=1
export VLLM_SKIP_WARMUP=true
export VLLM_EXPONENTIAL_BUCKETING=True
export VLLM_PROMPT_BS_BUCKET_MIN=1
export VLLM_PROMPT_BS_BUCKET_STEP=1
export VLLM_PROMPT_BS_BUCKET_MAX=64
export VLLM_PROMPT_SEQ_BUCKET_MIN=128
export VLLM_PROMPT_SEQ_BUCKET_STEP=1024
export VLLM_PROMPT_SEQ_BUCKET_MAX=32768
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800

vllm serve Qwen/Qwen3-Embedding-4B \
  --tensor-parallel-size 1 \
  --runner pooling \
  --convert embed \
  --max-model-len 32768 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.80 \
  --no-enable-prefix-caching \
  --no-enable-chunked-prefill \
  --hf-overrides '{"is_matryoshka": true}'
Run inside a vllm-gaudi container.
At a glance

Checks that passed

  • Retrieval nDCG@10 with the documented query instruction: SciFact 78.1, NFCorpus 40.9, ArguAna 73.5, CodeSearchNet python 95.9, QMSum 51.2
  • Multilingual Wikipedia retrieval nDCG@10: de 89.7, hi 86.8, pt 87.5; STSBenchmark Spearman 93.6
  • At 256 dimensions: SciFact 73.9, ArguAna 70.3, STSBenchmark 93.6
  • Stock image, no fixes: the checkpoint is plain Qwen3ForCausalLM and is mapped to Qwen3ForEmbedding at load

Notes

  • Queries need the prefix 'Instruct: <task>\nQuery:<query>'; documents go in bare. vLLM does not apply sentence-transformers prompts.
  • Cost per input is flat at about 50 ms whether inputs are batched or concurrent: pooling prefill runs one sequence at a time on this image. Scale out with replicas.
Guide

Qwen's 4B instruction-aware text embedding model (2,560 dimensions, Matryoshka truncation), served through vLLM's pooling runner in BF16 on a single Gaudi 2 card. Stock image, no fixes.
Model Qwen/Qwen3-Embedding-4B
Hardware 1x Gaudi 2 (TP 1)
Image vllm-gaudi v0.26.0 (b059f9c), vLLM 568afb3a, Gaudi SW 1.24.1
Precision BF16 (8 GB of weights)
Context 32,768 tokens per input
Concurrency --max-num-seqs 64
Endpoint /v1/embeddings
Recipe recipes/Qwen/Qwen3-Embedding-4B/recipe.yaml

Required fixes

None. The model has no vision tower, no MRoPE and no sampling, so none of the runtime fixes apply.

Client usage

The model is asymmetric. vLLM does not apply sentence-transformers prompt templates, so the client adds the query instruction; documents go in bare.

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:{query}

Embedding queries without the prefix, or documents with it, measurably lowers retrieval quality.

Verified

On vllm-gaudi v0.26.0, one Gaudi 2 card, with exactly the launch command above in a container started from the stock image, and the documented query instruction. Retrieval scores are nDCG@10, STS is Spearman; values in parentheses are at 256 dimensions.

Task Score
SciFact 78.1 (73.9)
NFCorpus 40.9 (35.5)
ArguAna 73.5 (70.3)
CodeSearchNet python 95.9 (95.0)
Wikipedia retrieval de / hi / pt 89.7 / 86.8 / 87.5
QMSum long documents 51.2 (44.9)
STSBenchmark 93.6 (93.6)

Performance

Single runs on the server above (measured as: per-call, one server, warm, inputs of about 20 tokens, best of 3).

Load Result
1 input, 1 request 58 ms
32 inputs in one request 1.69 s (53 ms per input)
256 inputs in one request 13.5 s (53 ms per input)

Known issues

  • Cost per input is flat at about 50 ms however the work arrives, the same as for Qwen3-Embedding-8B: pooling prefill runs one sequence at a time on this image, so the smaller model is not faster per input. Scale out with replicas.

What not to do

  • Do not mix vectors from this model (2,560 dimensions) and from Qwen3-Embedding-8B (4,096) in one index; truncate both to a common Matryoshka width if you must.
  • Do not add an instruction prefix to documents.