mirror of https://github.com/zenlm/engine.git synced 2026-10-30 15:00:05 +00:00

Files

T

Eric BuehlerandGitHub 272f7ee1b5 Add multi-model support! (#1512 )

* Refactor MistralRs

* Working multi-model!

* Add mutli-model docs initially

* Update mistralrs-pyo3, mistralrs-bench, mistralrs

* Update apis for consistency

* API tweaks

* Logging tweaks

* Add examples, tweak cli

* Clearer pipeline id

* Fix config key semantics

* Format and clippy

* Tweak logging, fix example

* Clippy refactor

* Update examples

* Remove unused multi model docs

* Replace 'ignore' with 'default'

* Update docs

2025-06-24 19:49:26 -04:00

1.1 KiB

Raw Permalink Blame History

Perplexity

What is peplexity? See: https://huggingface.co/docs/transformers/en/perplexity

To calculate perplexity over some text contained in a file, run:

cargo run --release --features ... --example perplexity -- --model-id ... --file ... --isq ...

For example,

wget https://huggingface.co/datasets/EricB/wikitext2/resolve/main/wiki.test_mini.raw
cargo run --release --features ... --example perplexity -- --model-id meta-llama/Llama-3.1-8B-Instruct --file wiki.test_mini.raw --isq 4

Note: A table of perplexity benchmarks will be coming soon!

Benchmark

All benchmarks on an M3 Max (Metal) with model: meta-llama/Llama-3.1-8B-Instruct, calibration data: calibration_datav3_small.txt (if applicable), and test data: wiki.test_small.raw.

Quantization	Mistral.rs Imatrix	Mistral.rs PPL	Mistral.rs ΔPPL
None	None	8.538557±2.3113003	0±0
Q6K	calibration_datav3_small.txt	8.559697±2.3130114	0.02114±0.0253533
Q5K	calibration_datav3_small.txt	8.599381±2.3180215	0.0608253±0.0438977
Q4K	calibration_datav3_small.txt	8.7962885±2.3615937	0.257732±0.0893235
Q3K	None	10.583125±2.794593	2.04457±0.586526

1.1 KiB Raw Permalink Blame History

Perplexity

Benchmark

1.1 KiB

Raw Permalink Blame History