Files
engine/docs/multi_model/advanced-config.json
Eric BuehlerandGitHub f0c70ecad5 Implement the new Google model (#2046)
* feat(models): implement the gemma4 family

* Some fixes to run

* Faster

* Handle think tags

* Handle think tags

* Further refactor

* Fix bugs

* Moe works

* Fix bos

* Fixes for vision

* Fix corner case

* Remove debugging

* Cleanup

* Further refinements

* Cleanup

* Cleanup rotating cache

* Fix cache

* Fix cache

* Fix mask and cache bug

* Fix prealloc bug

* Fix fa bug

* Fix small corner case

* Fix audios index

* MoE refactor

* More indexed moe quants

* Format

* Enable FA fallback

* Add paged attn support

* Fixes

* Fixes

* Audio works

* Audio works

* Audio fixes

* Fix default for reasoning

* Format

* Fixes for parity

* Fixes

* Fixes

* Format

* No isq for vision

* Fixes

* Remove vision isq

* Fix loading

* Fixes for vision moe

* Allowe looser case

* Allow default sampling params

* Fixes

* Fix

* Further fix

* Fix effective_sliding_window

* Fix bidirectional case

* Fix bidirectional FA

* Fix buffer zeroing

* Fix some nans

* Update

* Update rmsnorm

* Update cli

* Minor fixes

* Fix loading

* Add tool call support

* Fix tanh

* Update reasoning parser

* Update routing

* Update routing

* Attempt fix

* Attempt fix

* Attempt fix

* Fix search

* Cleanup

* candle 0.10.0

* Fix block table bug

* Remove unnecessary part

* Remove unnecessary part

* Remove unnecessary part

* Fixes

* Fixes

* Fixes

* vision -> multimodal

* Add video support

* Update docs

* Video fixes

* Video fixes

* Fix mask

* Add oneshot mode

* Add oneshot mode

* Fixes

* Update for full sdpa fix

* Rename thinking

* Better loading

* Add ffmpeg installation

* Dont re expand

* Fixes

* Fixes

* Minor tweaks

* Only image tokens get bidirectional

* Fix video prefix cache

* Better video handling

* Bidirectional on both

* Fix uqffs

* Allow auth for uqff

* Improve readme

* Fix uqff gen

* Update readme

* Fix uqff

* Fix uqff

* Disable paged

* Fix afq cuda

* Readme

* Readme

* Model card link

* Allow shorthand

* Bump versions

* Bump versions

* Bump versions

* Examples

* Update

* Docs

* Bump version

* Fix expand logic

* Bump cehck

* Bump cehck

* Bump
2026-04-02 12:06:57 -04:00

40 lines
918 B
JSON

{
"llama3-3b-text": {
"alias": "llama3-3b-text",
"Plain": {
"model_id": "meta-llama/Llama-3.2-3B-Instruct"
},
"in_situ_quant": "8"
},
"gemma4-E4B-multimodal": {
"alias": "gemma4-E4B-multimodal",
"MultimodalPlain": {
"model_id": "google/gemma-4-E4B-it"
},
"in_situ_quant": "4"
},
"qwen3-4b-fast": {
"alias": "qwen3-4b-fast",
"Plain": {
"model_id": "Qwen/Qwen3-4B"
},
"in_situ_quant": "4",
"num_device_layers": ["8"]
},
"llama3-gguf": {
"alias": "llama3-gguf",
"Gguf": {
"tok_model_id": "meta-llama/Llama-3.2-3B-Instruct",
"quantized_model_id": "bartowski/Llama-3.2-3B-Instruct-GGUF",
"quantized_filename": "Llama-3.2-3B-Instruct-Q4_K_M.gguf"
}
},
"embed-qwen3": {
"alias": "embed-qwen3",
"Embedding": {
"model_id": "Qwen/Qwen3-Embedding-0.6B",
"arch": "qwen3embedding"
}
}
}