Files
skills/public/modal-gpu-workloads.html
zeekay 747dafba10 Register WebMCP tools on every page, and name each page's twin from site.py
Every page now loads /webmcp.js, scripts/webmcp.ts with its types
stripped (npm run webmcp). Where the browser has document.modelContext
it registers two read-only tools: search_skills over the published
skills index, and get_skill, which returns a skill's markdown checked
against the index digest. The markdown alternate link moves from
build.py to site.py, so CI holds every page to it.

The API catalog gains the site itself (llms.txt and the skills index)
and hanzo.ai/agents; the live discovery check now fetches every
catalog link. tests/site.mjs drives the tools through a recording
modelContext, and checks a browser without one loads clean.
2026-10-26 20:57:05 -07:00

627 lines
26 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>modal-gpu-workloads</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/modal-gpu-workloads">
<link rel="alternate" type="text/markdown" href="/modal-gpu-workloads.md">
<script type="module" src="/webmcp.js"></script>
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/modal-gpu-workloads.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>modal-gpu-workloads</h1><p class="lede">Running ML/AI inference on Modal</p><h1>Modal GPU Workloads</h1>
<p><strong>Use this skill when:</strong></p>
<ul><li>Running ML/AI inference on Modal</li><li>Training models on GPU</li><li>Selecting appropriate GPU types</li><li>Optimizing GPU utilization and costs</li><li>Working with PyTorch, TensorFlow, or JAX</li></ul>
<h2>GPU Selection</h2>
<h3>Available GPU Types</h3>
<p>Choose GPU based on workload:</p>
<pre><code class="lang-python">import modal
app = modal.App(&quot;gpu-app&quot;)
# T4 - Budget-friendly, good for inference
@app.function(gpu=&quot;t4&quot;)
def t4_inference(text: str):
# 16GB VRAM, ~$0.50/hour
return run_small_model(text)
# L4 - Good balance for most workloads
@app.function(gpu=&quot;l4&quot;)
def l4_inference(text: str):
# 24GB VRAM, ~$1.00/hour
return run_medium_model(text)
# L40S - RECOMMENDED for most ML workloads
# Best cost/performance ratio
@app.function(gpu=&quot;l40s&quot;)
def l40s_inference(text: str):
# 48GB VRAM, great price/performance
return run_large_model(text)
# A100 - High-end for large models/training
@app.function(gpu=&quot;a100&quot;)
def a100_training():
# 40GB or 80GB VRAM, ~$4-8/hour
return train_large_model()
# H100 - Cutting edge for very large workloads
@app.function(gpu=&quot;h100&quot;)
def h100_training():
# 80GB VRAM, ~$8-12/hour
return train_huge_model()
# Multiple GPUs
@app.function(gpu=&quot;a100:4&quot;) # 4x A100
def multi_gpu_training():
return distributed_training()</code></pre>
<h3>GPU Recommendations</h3>
<p><strong>For inference:</strong></p>
<ul><li>Small models (&lt;7B params): T4 or L4</li><li>Medium models (7-13B params): L40S</li><li>Large models (13-70B params): L40S or A100</li><li>Very large models (&gt;70B params): A100 or H100</li></ul>
<p><strong>For training:</strong></p>
<ul><li>Fine-tuning small models: L4 or L40S</li><li>Fine-tuning medium models: L40S or A100</li><li>Training from scratch: A100 or H100</li><li>Distributed training: Multiple A100 or H100</li></ul>
<h2>PyTorch on GPU</h2>
<h3>Basic Inference</h3>
<p>Set up PyTorch with GPU:</p>
<pre><code class="lang-python">image = (
modal.Image.debian_slim()
.uv_pip_install(
&quot;torch==2.1.0&quot;,
&quot;transformers==4.35.0&quot;
)
)
@app.cls(gpu=&quot;l40s&quot;, image=image)
class TextGenerator:
@modal.enter()
def load_model(self):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = &quot;meta-llama/Llama-2-7b-hf&quot;
print(f&quot;Loading model {model_name}...&quot;)
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
device_map=&quot;auto&quot;
)
print(&quot;Model loaded&quot;)
@modal.method()
def generate(self, prompt: str, max_length: int = 100) -&gt; str:
import torch
inputs = self.tokenizer(prompt, return_tensors=&quot;pt&quot;).to(&quot;cuda&quot;)
with torch.no_grad():
outputs = self.model.generate(
**inputs,
max_length=max_length,
temperature=0.7,
do_sample=True
)
return self.tokenizer.decode(outputs[0], skip_special_tokens=True)
@app.local_entrypoint()
def main():
generator = TextGenerator()
result = generator.generate.remote(&quot;Once upon a time&quot;)
print(result)</code></pre>
<h3>Batch Inference</h3>
<p>Process multiple inputs efficiently:</p>
<pre><code class="lang-python">@app.cls(gpu=&quot;l40s&quot;, image=image)
class BatchInference:
@modal.enter()
def load_model(self):
import torch
from transformers import pipeline
self.classifier = pipeline(
&quot;sentiment-analysis&quot;,
model=&quot;distilbert-base-uncased-finetuned-sst-2-english&quot;,
device=0 # GPU 0
)
@modal.method()
def predict_batch(self, texts: list[str]) -&gt; list[dict]:
# Process in batches for efficiency
batch_size = 32
results = []
for i in range(0, len(texts), batch_size):
batch = texts[i:i + batch_size]
batch_results = self.classifier(batch)
results.extend(batch_results)
return results
@app.local_entrypoint()
def main():
texts = [&quot;I love this!&quot; for _ in range(1000)]
classifier = BatchInference()
results = classifier.predict_batch.remote(texts)
print(f&quot;Processed {len(results)} texts&quot;)</code></pre>
<h2>Model Optimization</h2>
<h3>Quantization</h3>
<p>Reduce memory usage with quantization:</p>
<pre><code class="lang-python">image = (
modal.Image.debian_slim()
.uv_pip_install(
&quot;torch==2.1.0&quot;,
&quot;transformers==4.35.0&quot;,
&quot;bitsandbytes==0.41.0&quot;,
&quot;accelerate==0.24.0&quot;
)
)
@app.cls(gpu=&quot;l40s&quot;, image=image)
class QuantizedModel:
@modal.enter()
def load_model(self):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
# 4-bit quantization config
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type=&quot;nf4&quot;
)
self.tokenizer = AutoTokenizer.from_pretrained(&quot;meta-llama/Llama-2-7b-hf&quot;)
self.model = AutoModelForCausalLM.from_pretrained(
&quot;meta-llama/Llama-2-7b-hf&quot;,
quantization_config=quantization_config,
device_map=&quot;auto&quot;
)
@modal.method()
def generate(self, prompt: str) -&gt; str:
inputs = self.tokenizer(prompt, return_tensors=&quot;pt&quot;).to(&quot;cuda&quot;)
outputs = self.model.generate(**inputs, max_length=100)
return self.tokenizer.decode(outputs[0])</code></pre>
<h3>Flash Attention</h3>
<p>Use Flash Attention for faster inference:</p>
<pre><code class="lang-python">image = (
modal.Image.debian_slim()
.uv_pip_install(
&quot;torch==2.1.0&quot;,
&quot;transformers==4.35.0&quot;,
&quot;flash-attn==2.3.3&quot;
)
)
@app.cls(gpu=&quot;l40s&quot;, image=image)
class FastModel:
@modal.enter()
def load_model(self):
import torch
from transformers import AutoModelForCausalLM
self.model = AutoModelForCausalLM.from_pretrained(
&quot;meta-llama/Llama-2-7b-hf&quot;,
torch_dtype=torch.float16,
device_map=&quot;auto&quot;,
attn_implementation=&quot;flash_attention_2&quot;
)</code></pre>
<h2>Training on GPU</h2>
<h3>Fine-Tuning</h3>
<p>Fine-tune models on GPU:</p>
<pre><code class="lang-python">image = (
modal.Image.debian_slim()
.uv_pip_install(
&quot;torch==2.1.0&quot;,
&quot;transformers==4.35.0&quot;,
&quot;datasets==2.14.0&quot;,
&quot;accelerate==0.24.0&quot;
)
)
@app.function(
gpu=&quot;a100&quot;,
timeout=3600, # 1 hour
image=image
)
def fine_tune_model(dataset_name: str, output_path: str):
import torch
from transformers import (
AutoModelForSequenceClassification,
AutoTokenizer,
Trainer,
TrainingArguments
)
from datasets import load_dataset
# Load model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained(
&quot;distilbert-base-uncased&quot;,
num_labels=2
).to(&quot;cuda&quot;)
tokenizer = AutoTokenizer.from_pretrained(&quot;distilbert-base-uncased&quot;)
# Load and prepare dataset
dataset = load_dataset(dataset_name)
def tokenize(examples):
return tokenizer(
examples[&quot;text&quot;],
padding=&quot;max_length&quot;,
truncation=True
)
tokenized_dataset = dataset.map(tokenize, batched=True)
# Training configuration
training_args = TrainingArguments(
output_dir=output_path,
num_train_epochs=3,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
warmup_steps=500,
weight_decay=0.01,
logging_dir=&quot;./logs&quot;,
fp16=True, # Mixed precision training
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset[&quot;train&quot;],
eval_dataset=tokenized_dataset[&quot;test&quot;]
)
# Train
trainer.train()
# Save model
trainer.save_model(output_path)
return {&quot;status&quot;: &quot;complete&quot;, &quot;output&quot;: output_path}</code></pre>
<h3>Distributed Training</h3>
<p>Train across multiple GPUs:</p>
<pre><code class="lang-python">@app.function(
gpu=&quot;a100:4&quot;, # 4 GPUs
timeout=7200,
image=image
)
def distributed_training():
import torch
import torch.distributed as dist
from transformers import Trainer, TrainingArguments
# Training args for distributed training
training_args = TrainingArguments(
output_dir=&quot;./output&quot;,
num_train_epochs=3,
per_device_train_batch_size=8,
gradient_accumulation_steps=4,
fp16=True,
deepspeed=&quot;ds_config.json&quot;, # DeepSpeed config
local_rank=-1 # Auto-set by launcher
)
# Trainer handles distribution automatically
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset
)
trainer.train()</code></pre>
<h2>GPU Memory Management</h2>
<h3>Monitoring Memory</h3>
<p>Track GPU memory usage:</p>
<pre><code class="lang-python">@app.function(gpu=&quot;l40s&quot;)
def monitor_memory():
import torch
print(f&quot;Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB&quot;)
print(f&quot;Reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB&quot;)
print(f&quot;Max allocated: {torch.cuda.max_memory_allocated() / 1e9:.2f} GB&quot;)
# Clear cache
torch.cuda.empty_cache()
return &quot;Memory checked&quot;</code></pre>
<h3>Gradient Checkpointing</h3>
<p>Reduce memory for training:</p>
<pre><code class="lang-python">@app.function(gpu=&quot;l40s&quot;, image=image)
def train_with_checkpointing():
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(&quot;gpt2&quot;)
# Enable gradient checkpointing
model.gradient_checkpointing_enable()
# Train with reduced memory
# (slower but fits larger models)</code></pre>
<h2>Cost Optimization</h2>
<h3>Auto-Shutdown</h3>
<p>Set timeouts to avoid unnecessary costs:</p>
<pre><code class="lang-python"># Timeout after 5 minutes of inactivity
@app.function(gpu=&quot;l40s&quot;, timeout=300)
def inference_with_timeout(text: str):
return generate(text)
# Container idle timeout
@app.function(
gpu=&quot;l40s&quot;,
container_idle_timeout=60 # Shutdown after 60s idle
)
def auto_shutdown_inference(text: str):
return generate(text)</code></pre>
<h3>Warm Pool</h3>
<p>Keep containers warm for low latency:</p>
<pre><code class="lang-python"># Keep 1 container warm for instant response
@app.function(
gpu=&quot;l40s&quot;,
keep_warm=1
)
def low_latency_inference(text: str):
return generate(text)</code></pre>
<h2>Testing GPU Code Locally</h2>
<h3>CPU Fallback for Development</h3>
<p>Test without GPU:</p>
<pre><code class="lang-python">@app.function(gpu=&quot;l40s&quot; if modal.is_remote() else None)
def flexible_inference(text: str):
import torch
device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;
print(f&quot;Using device: {device}&quot;)
model = load_model().to(device)
return model(text)</code></pre>
<h2>Anti-Patterns to Avoid</h2>
<p><strong>DON&#x27;T load models on every invocation:</strong></p>
<pre><code class="lang-python"># ❌ BAD - Loads model each time (very slow)
@app.function(gpu=&quot;l40s&quot;)
def bad_inference(text: str):
model = load_model() # Slow!
return model(text)
# ✅ GOOD - Load once per container
@app.cls(gpu=&quot;l40s&quot;)
class GoodInference:
@modal.enter()
def load_model(self):
self.model = load_model()
@modal.method()
def predict(self, text: str):
return self.model(text)</code></pre>
<p><strong>DON&#x27;T use expensive GPUs for simple tasks:</strong></p>
<pre><code class="lang-python"># ❌ BAD - H100 for tiny model
@app.function(gpu=&quot;h100&quot;)
def overkill(text: str):
return simple_model(text) # Wastes $10/hour
# ✅ GOOD - Match GPU to workload
@app.function(gpu=&quot;t4&quot;)
def appropriate(text: str):
return simple_model(text) # Costs $0.50/hour</code></pre>
<p><strong>DON&#x27;T forget to set timeouts:</strong></p>
<pre><code class="lang-python"># ❌ BAD - Could run forever
@app.function(gpu=&quot;a100&quot;)
def no_timeout():
while True:
process()
# ✅ GOOD - Bounded execution
@app.function(gpu=&quot;a100&quot;, timeout=3600)
def with_timeout():
process_for_limited_time()</code></pre>
<h2>Related Skills</h2>
<ul><li><strong>modal-functions-basics.md</strong> - Basic Modal function patterns</li><li><strong>modal-image-building.md</strong> - Installing ML dependencies</li><li><strong>modal-volumes-secrets.md</strong> - Storing models and data</li><li><strong>modal-web-endpoints.md</strong> - Serving models via HTTP</li></ul>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/modal-gpu-workloads.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>