Files
zeekayandHanzo Dev 6a363f7ee3 merge: upstream main ddf1b879 and the aarch64 repack branch 35d7ae7c
Three-way per file through the rename: base 39355c6c91, theirs the upstream
merge of main with 35d7ae7c (via 7ef6a694d), ours origin/main.

Lands the barrier thread pool, runtime CPU feature dispatch, aarch64 and x86
repacked quantized kernels (indexed_gemv, gemv_fused_shared_lhs), quantized
embedding on CPU/Metal/CUDA, paged flash-attn kernels, cutile fused MoE, the
in-place op aliasing API, parking_lot storage locks, Metal buffer builders and
debug labels, safetensors 0.8, hf-hub 1.0, tokenizers 0.23.

Ours keeps ROCm, Vulkan, wgpu, the IQ/MXFP4/NVFP4/TQ dtypes, and the i-quant
weight/activation block split in matmul. Our backends ride InplaceOpN and the
quantized embedding; Metal get_rows gains the IQ kernels. ug leaves, as upstream.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-15 15:52:28 -07:00
..

hanzo-ml-stella-en-v5: Implementation of stella_en_1.5B_v5 embedding model

As of 7th Oct 2024, Stella_en_1.5B_v5 is one of the top ranking model on retrieval and reranking tasks in MTEB leaderboard.

Model card on the HuggingFace Hub.

Running the example

Stella_en_1.5B_v5 is used to generate text embeddings for a prompt. The model weights are downloaded from the hub on the first run.

$ cargo run --example stella-en-v5 --release  -- --query "What are safetensors?" --which 1.5b

> [[ 0.3905, -0.0130,  0.2072, ..., -0.1100, -0.0086,  0.6002]]
>  Tensor[[1, 1024], f32]

Stella_en_1.5B_v5 is trained by MRL enabling multiple embedding dimensions.

The following reproduces the example in the model card for a retrieval task (s2p). The sample queries and docs are hardcoded in the example.

$ cargo run --example stella-en-v5 --release --features <metal | cuda> -- --which 1.5b

>
> Score: 0.8178786
> Query: What are some ways to reduce stress?
> Answer: There are many effective ways to reduce stress. Some common techniques include deep breathing, meditation, and physical activity. Engaging in hobbies, spending
> time in nature, and connecting with loved ones can also help alleviate stress. Additionally, setting boundaries, practicing self-care, and learning to say no can prevent
> stress from building up.
>
>
> Score: 0.7853528
> Query: What are the benefits of drinking green tea?
> Answer: Green tea has been consumed for centuries and is known for its potential health benefits. It contains antioxidants that may help protect the body against damage 
> caused by free radicals. Regular consumption of green tea has been associated with improved heart health, enhanced cognitive function, and a reduced risk of certain types >
> of cancer. The polyphenols in green tea may also have anti-inflammatory and weight loss properties.
>

$ cargo run --example stella-en-v5 --release --features <metal | cuda> -- --which 400m

>
> Score: 0.8397539
> Query: What are some ways to reduce stress?
> Answer: There are many effective ways to reduce stress. Some common techniques include deep breathing, meditation, and physical activity. Engaging in hobbies, spending
> time in nature, and connecting with loved ones can also help alleviate stress. Additionally, setting boundaries, practicing self-care, and learning to say no can prevent
> stress from building up.
>
>
>
> Score: 0.809545
> Query: What are the benefits of drinking green tea?
> Answer: Green tea has been consumed for centuries and is known for its potential health benefits. It contains antioxidants that may help protect the body against damage
> caused by free radicals. Regular consumption of green tea has been associated with improved heart health, enhanced cognitive function, and a reduced risk of certain types
> of cancer. The polyphenols in green tea may also have anti-inflammatory and weight loss properties.
>

Supported options:

  • Stella_en_v5 has 2 model variants published - a 1.5B variant and 400M variant. This is enabled through the flag --which. E.g. --which 400m or --which 1.5b.

  • Stella_en_v5 supports 256, 768, 1024, 2048, 4096, 6144 and 8192 embedding dimensions (though the model card mentions 512, I couldn't find weights for the same). In the example run this is supported with --embed-dim option. E.g. ... --embed-dim 4096. Defaults to 1024.

  • As per the model card, the model has been primarily trained on s2s (similarity) and s2p (retrieval) tasks. These require a slightly different query preprocessing (a different prompt template for each). In this example this is enabled though --task option.