Files
engine/examples/multi-model-advanced-config.json
Eric BuehlerandGitHub f0c70ecad5 Implement the new Google model (#2046)
* feat(models): implement the gemma4 family

* Some fixes to run

* Faster

* Handle think tags

* Handle think tags

* Further refactor

* Fix bugs

* Moe works

* Fix bos

* Fixes for vision

* Fix corner case

* Remove debugging

* Cleanup

* Further refinements

* Cleanup

* Cleanup rotating cache

* Fix cache

* Fix cache

* Fix mask and cache bug

* Fix prealloc bug

* Fix fa bug

* Fix small corner case

* Fix audios index

* MoE refactor

* More indexed moe quants

* Format

* Enable FA fallback

* Add paged attn support

* Fixes

* Fixes

* Audio works

* Audio works

* Audio fixes

* Fix default for reasoning

* Format

* Fixes for parity

* Fixes

* Fixes

* Format

* No isq for vision

* Fixes

* Remove vision isq

* Fix loading

* Fixes for vision moe

* Allowe looser case

* Allow default sampling params

* Fixes

* Fix

* Further fix

* Fix effective_sliding_window

* Fix bidirectional case

* Fix bidirectional FA

* Fix buffer zeroing

* Fix some nans

* Update

* Update rmsnorm

* Update cli

* Minor fixes

* Fix loading

* Add tool call support

* Fix tanh

* Update reasoning parser

* Update routing

* Update routing

* Attempt fix

* Attempt fix

* Attempt fix

* Fix search

* Cleanup

* candle 0.10.0

* Fix block table bug

* Remove unnecessary part

* Remove unnecessary part

* Remove unnecessary part

* Fixes

* Fixes

* Fixes

* vision -> multimodal

* Add video support

* Update docs

* Video fixes

* Video fixes

* Fix mask

* Add oneshot mode

* Add oneshot mode

* Fixes

* Update for full sdpa fix

* Rename thinking

* Better loading

* Add ffmpeg installation

* Dont re expand

* Fixes

* Fixes

* Minor tweaks

* Only image tokens get bidirectional

* Fix video prefix cache

* Better video handling

* Bidirectional on both

* Fix uqffs

* Allow auth for uqff

* Improve readme

* Fix uqff gen

* Update readme

* Fix uqff

* Fix uqff

* Disable paged

* Fix afq cuda

* Readme

* Readme

* Model card link

* Allow shorthand

* Bump versions

* Bump versions

* Bump versions

* Examples

* Update

* Docs

* Bump version

* Fix expand logic

* Bump cehck

* Bump cehck

* Bump
2026-04-02 12:06:57 -04:00

73 lines
1.7 KiB
JSON

[
{
"model_id": "llama3-3b-instruct",
"model": {
"Plain": {
"model_id": "meta-llama/Llama-3.2-3B-Instruct"
}
},
"in_situ_quant": "4"
},
{
"model_id": "qwen3-4b",
"model": {
"Plain": {
"model_id": "Qwen/Qwen3-4B"
}
},
"num_device_layers": ["8"],
"in_situ_quant": "4"
},
{
"model_id": "qwen2-vl",
"model": {
"MultimodalPlain": {
"model_id": "Qwen/Qwen2-VL-2B-Instruct"
}
}
},
{
"model_id": "gemma4-E4B-multimodal",
"model": {
"MultimodalPlain": {
"model_id": "google/gemma-4-E4B-it"
}
}
},
{
"model_id": "phi3-multimodal",
"model": {
"MultimodalPlain": {
"model_id": "microsoft/Phi-3-vision-128k-instruct"
}
}
},
{
"model_id": "llama-gguf-q4",
"model": {
"Gguf": {
"tok_model_id": null,
"quantized_model_id": "TheBloke/Llama-2-7b-Chat-GGUF",
"quantized_filename": "llama-2-7b-chat.Q4_K_M.gguf"
}
}
},
{
"model_id": "deepseek-coder",
"model": {
"Plain": {
"model_id": "deepseek-ai/deepseek-coder-1.3b-instruct"
}
},
"chat_template": "{%- if messages[0]['role'] == 'system' -%}{{ messages[0]['content'] }}\n{%- endif -%}{%- for message in messages -%}{%- if message['role'] == 'user' -%}### Instruction:\n{{ message['content'] }}\n\n### Response:\n{%- elif message['role'] == 'assistant' -%}{{ message['content'] }}\n{%- endif -%}{%- endfor -%}"
},
{
"model_id": "flux-schnell",
"model": {
"DiffusionPlain": {
"model_id": "black-forest-labs/FLUX.1-schnell",
"dtype": "bf16"
}
}
}
]