mirror of
https://github.com/luxfhe/ml-sdk.git
synced 2026-10-30 18:13:04 +00:00
zeekay
bde2e2531b
Machine Learning with FHE - fork of Zama Concrete-ML - Built-in FHE-friendly ML models - scikit-learn compatible API - PyTorch model conversion - Quantization-aware training
1.0 KiB
1.0 KiB
Hybrid model
This use case example showcases how to partially run layers in FHE.
In this case we apply a fully connected layer of a GPT-2 model in FHE.
How to run this use-case
- Install additional requirements using
python -m pip install -r requirements.txt - Compile GPT-2 model using
python compile_hybrid_llm.pyscript - Run FHE server using
bash serve.sh - Run FHE client using
python infer_hybrid_llm_generate.py- You will first be asked about the number of tokens that you want to generate
- Then you will be able to enter your prompt
Adapt to any LLM
This use case can easily be used with a different model than GPT2 with any other model.
For example, you can use Phi 1.5 on Hugging Face and protect the weights that are used to compute Query, Keys, Values as follows:
python compile_hybrid_llm.py --model-name microsoft/phi-1_5 --module-names model.layers.0.self_attn.q_proj,model.layers.0.self_attn.k_proj,model.layers.0.self_attn.v_proj