mirror of
https://github.com/zenlm/engine.git
synced 2026-10-30 15:00:05 +00:00
Eric BuehlerandGitHub
ff2ccd511d
* Sketch the diffusion pipeline, loader, model, and FLUX model * Connect response part to openai server * Sketch the flux loader * Handle 256 limitiation of timestep distilliation model * Work on loading * Handle raw prompts for image gen models * Should build now * Clippy * Add diffusion plain model * Typos * Sketch a server example * Add parse_diffusion_arch * Add debug bound * Use as_any * Some fixes * Fix config * Loads now * Avoid empty prompt err * Fix overflow * Fix overflow * Use the correct processor * Add debugs * Cast to correct dtype * Squeeze * Use sharded and downcast repo * Better loading * Dont modify cache * Fix response * Use big t5 for now * Only load relevant tensors * Clamp for f16 * Do some things * Use sharded and downcast repo * Diffusion model case * Oops * Oops * Hotload t5xxl * Use only encoder parts * Don't cast it * Fix files * Format * Cast every layer * Updated resolution and prompt * Fix issue with dual running * Reduce lifetime of hotloaded t5 xxl * Offloading t5 layers to cpu * Add offloading config * Add Python, Rust support, docs * Add configurable height and width
20 lines
432 B
Python
20 lines
432 B
Python
from mistralrs import (
|
|
Runner,
|
|
Which,
|
|
DiffusionArchitecture,
|
|
ImageGenerationResponseFormat,
|
|
)
|
|
|
|
runner = Runner(
|
|
which=Which.DiffusionPlain(
|
|
model_id="mistralai/Mistral-7B-Instruct-v0.1",
|
|
arch=DiffusionArchitecture.FluxOffloaded,
|
|
),
|
|
)
|
|
|
|
res = runner.generate_image(
|
|
"A vibrant sunset in the mountains, 4k, high quality.",
|
|
ImageGenerationResponseFormat.Url,
|
|
)
|
|
print(res.choices[0].url)
|