Three-way per file through the rename: base 39355c6c91, theirs the upstream
merge of main with 35d7ae7c (via 7ef6a694d), ours origin/main.
Lands the barrier thread pool, runtime CPU feature dispatch, aarch64 and x86
repacked quantized kernels (indexed_gemv, gemv_fused_shared_lhs), quantized
embedding on CPU/Metal/CUDA, paged flash-attn kernels, cutile fused MoE, the
in-place op aliasing API, parking_lot storage locks, Metal buffer builders and
debug labels, safetensors 0.8, hf-hub 1.0, tokenizers 0.23.
Ours keeps ROCm, Vulkan, wgpu, the IQ/MXFP4/NVFP4/TQ dtypes, and the i-quant
weight/activation block split in matmul. Our backends ride InplaceOpN and the
quantized embedding; Metal get_rows gains the IQ kernels. ug leaves, as upstream.
Co-authored-by: Hanzo Dev <[email protected]>
NV-Embed-v2
ML implementation (inference only) of NV-Embed-v2, a text embedding model that ranks No. 1 (as of Nov 25 2024) on the MTEB benchmark with a score of 72.31 across 56 text embedding tasks.
Running an example: Retrieval
cargo run --example nvembed_v2 --release
> scores: [[87.4269, 0.4629],
> [ 0.9653, 86.0372]]
> Tensor[[2, 2], f32]
In this example, we have two queries and two passages (the corresponding answers). The output tensor represents the similarity scores between each query-passage pair. The scores are computed by taking the dot product of the query and passage embeddings and scaling the result by 100.
let queries = [
"are judo throws allowed in wrestling?",
"how to become a radiology technician in michigan?",
];
let query_instruction =
"Instruct: Given a question, retrieve passages that answer the question\nQuery: "
.to_string();
let passages = [
"Since you're reading this, you are probably someone from a judo background or someone who is just wondering how judo techniques can be applied under wrestling rules. So without further ado, let's get to the question. Are Judo throws allowed in wrestling? Yes, judo throws are allowed in freestyle and folkstyle wrestling. You only need to be careful to follow the slam rules when executing judo throws. In wrestling, a slam is lifting and returning an opponent to the mat with unnecessary force.",
"Below are the basic steps to becoming a radiologic technologist in Michigan:Earn a high school diploma. As with most careers in health care, a high school education is the first step to finding entry-level employment. Taking classes in math and science, such as anatomy, biology, chemistry, physiology, and physics, can help prepare students for their college studies and future careers.Earn an associate degree. Entry-level radiologic positions typically require at least an Associate of Applied Science. Before enrolling in one of these degree programs, students should make sure it has been properly accredited by the Joint Review Committee on Education in Radiologic Technology (JRCERT).Get licensed or certified in the state of Michigan."
];
let passage_instruction = "".to_string();
If you already have the model and tokenizer files, you can use the --tokenizer and --model-files options to specify their full paths, instead of downloading them from the hub.
Running an example: Sentence embedding
cargo run --example nvembed_v2 --release -- --prompt "Here is a test sentence"
> Embedding: [[ 0.0066, -0.0048, 0.0066, ..., -0.0096, 0.0119, -0.0052]]
> Tensor[[1, 4096], f32]
In this example, we pass a prompt to the model and it outputs the vector encoding of the prompt.
Hardware Requirements
29.25GB at fp32
License
CC-BY-NC-4.0. This model should not be used for any commercial purpose. Refer the license for the detailed terms.