mirror of https://github.com/zenlm/zen5-engine.git synced 2026-10-30 14:57:31 +00:00

Files

T

antirez d0400a15bd Add DS4 imatrix and GGUF quantization tools

Add routed-MoE imatrix collection to the DS4 graph path, including dataset generation docs and the tracked calibration prompt corpus.

Add a standalone DeepSeek V4 Flash HF-to-GGUF quantizer that reuses GGUF template metadata, emits DS4 Q2/Q4 recipes, supports imatrix-aware routed expert quantization, and internalizes the required Q8_0, Q4_K, Q2_K, and IQ2_XXS quantizers instead of vendoring GGML.

Document quantizer provenance, output formats, and the DS4-specific imatrix mapping used by the generated GGUF files.

2026-05-12 18:34:08 +02:00

1.9 KiB

Raw Permalink Blame History

Official-Continuation Quality Testing

This directory contains the 100 prompts and scripts used to compare local GGUF variants against official DeepSeek V4 Flash continuations.

The metric is target-token negative log likelihood: collect a deterministic official continuation, then ask each local GGUF how much probability it assigns to that exact continuation token by token. This avoids judging quality from one sampled answer.

1. Collect Official Continuations

export DEEPSEEK_API_KEY=...
python3 gguf-tools/quality-testing/collect_official.py \
  --prompts gguf-tools/quality-testing/prompts.jsonl \
  --out gguf-tools/quality-testing/data \
  --count 100 \
  --max-tokens 24

The script writes:

data/prompts/case_*.txt
data/continuations/case_*.txt
data/responses/case_*.json
data/manifest.tsv

The prompt list is tracked in prompts.jsonl; the official responses are not tracked because they are derived from an external API.

2. Build The Local Scorer

make -C gguf-tools quality-score

The scorer links against the DS4 runtime and uses Metal by default.

3. Score GGUF Variants

gguf-tools/quality-testing/score_official \
  ../deepseek-v4-quants/gguf/OLD.gguf \
  gguf-tools/quality-testing/data/manifest.tsv \
  /tmp/old.tsv \
  4096

gguf-tools/quality-testing/score_official \
  ../deepseek-v4-quants/gguf/NEW.gguf \
  gguf-tools/quality-testing/data/manifest.tsv \
  /tmp/new.tsv \
  4096

4. Compare

python3 gguf-tools/quality-testing/compare_scores.py /tmp/old.tsv /tmp/new.tsv

Output fields:

avg_nll: average negative log likelihood; lower is better.
delta_new_minus_old: negative means the new GGUF fits the official continuation better.
case_wins_new_old_ties: per-prompt NLL wins.
first_token_matches: how often the local greedy first token matches the official first token.
avg_greedy_lcp: average greedy longest common prefix against the official continuation.

1.9 KiB Raw Permalink Blame History