Files
skills/public/error-budget-policy.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

395 lines
26 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>error-budget-policy</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/error-budget-policy.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/error-budget-policy">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/error-budget-policy.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>error-budget-policy</h1><p class="lede">Service: Payment API Owner: Payments Team Last Updated: 2025-10-27 Version: 2.0</p><h1>Error Budget Policy</h1>
<p><strong>Service</strong>: Payment API <strong>Owner</strong>: Payments Team <strong>Last Updated</strong>: 2025-10-27 <strong>Version</strong>: 2.0</p>
<h2>Overview</h2>
<p>This document defines how we use error budgets to balance reliability and feature velocity. The error budget is the allowed amount of unreliability, calculated from our SLO targets.</p>
<h2>Error Budget Calculation</h2>
<h3>Current SLOs</h3>
<p>| Metric | Target | Error Budget | |--------|--------|--------------| | Availability | 99.95% | 0.05% = 21.6 minutes/month | | Latency (p95) | &lt; 300ms | 5% requests may exceed | | Latency (p99) | &lt; 1s | 1% requests may exceed |</p>
<h3>Time-Based Budget</h3>
<pre><code>Total time in 30 days: 43,200 minutes
Error budget: 0.05% = 21.6 minutes
Budget per week: ~5.4 minutes
Budget per day: ~0.72 minutes (43 seconds)</code></pre>
<h2>Budget Consumption Zones</h2>
<h3>Zone 1: Healthy (&lt; 25% consumed)</h3>
<p><strong>Status</strong>: 🟢 Green</p>
<p><strong>Characteristics</strong>:</p>
<ul><li>Less than 5.4 minutes of downtime consumed</li><li>System operating well within SLO</li><li>Plenty of budget for innovation</li></ul>
<p><strong>Actions</strong>:</p>
<ul><li><strong>Development</strong>: Normal velocity, full feature work</li><li><strong>Releases</strong>: Continuous deployment enabled</li><li><strong>Changes</strong>: No restrictions, experiment freely</li><li><strong>Monitoring</strong>: Standard dashboards and alerts</li></ul>
<p><strong>Meeting Cadence</strong>: Monthly SLO review</p>
<p><strong>Approval Required</strong>: No</p>
<hr>
<h3>Zone 2: Concerning (25-50% consumed)</h3>
<p><strong>Status</strong>: 🟡 Yellow</p>
<p><strong>Characteristics</strong>:</p>
<ul><li>5.4 - 10.8 minutes consumed</li><li>Elevated risk of SLO miss</li><li>Need to be more careful with changes</li></ul>
<p><strong>Actions</strong>:</p>
<ul><li><strong>Development</strong>: Add reliability tasks to sprint backlog</li><li><strong>Releases</strong>: Staged rollouts with extended monitoring</li><li><strong>Changes</strong>: Risk assessment required for risky changes</li><li><strong>Monitoring</strong>: Increase alert sensitivity, daily checks</li></ul>
<p><strong>Restrictions</strong>:</p>
<ul><li>Defer non-critical features</li><li>No breaking changes without careful review</li><li>Extended canary periods (50% → 100% over 2 hours)</li></ul>
<p><strong>Meeting Cadence</strong>: Bi-weekly SLO review</p>
<p><strong>Approval Required</strong>: No, but Tech Lead should be informed</p>
<p><strong>Communication</strong>: Notify team in #payments-sre Slack channel</p>
<hr>
<h3>Zone 3: Critical (50-75% consumed)</h3>
<p><strong>Status</strong>: 🟠 Orange</p>
<p><strong>Characteristics</strong>:</p>
<ul><li>10.8 - 16.2 minutes consumed</li><li>High risk of missing SLO</li><li>Urgent reliability focus needed</li></ul>
<p><strong>Actions</strong>:</p>
<ul><li><strong>Development</strong>: 50% sprint capacity on reliability improvements</li><li><strong>Releases</strong>: Manual approval required for all deployments</li><li><strong>Changes</strong>: Feature freeze on non-critical changes</li><li><strong>Monitoring</strong>: Daily SLO review meetings, extended on-call</li></ul>
<p><strong>Restrictions</strong>:</p>
<ul><li><strong>Feature freeze</strong>: Only critical features and bug fixes</li><li><strong>All changes require approval</strong> from Tech Lead AND Engineering Manager</li><li><strong>Mandatory postmortems</strong> for all incidents</li><li><strong>Extended canary</strong>: 5% → 25% → 50% → 100% over 6 hours minimum</li></ul>
<p><strong>Required Actions</strong>:</p>
<ol><li>Identify top sources of error budget consumption</li><li>Create action items to reduce toil and improve reliability</li><li>Review recent changes for potential rollback</li><li>Assess if recent features should be rolled back or fixed</li></ol>
<p><strong>Meeting Cadence</strong>: Daily standup focused on SLO</p>
<p><strong>Approval Required</strong>: Yes (Tech Lead + Engineering Manager)</p>
<p><strong>Communication</strong>:</p>
<ul><li>Slack: #payments-sre, #engineering-all</li><li>Email: [email protected]</li><li>Escalation: VP Engineering notified</li></ul>
<hr>
<h3>Zone 4: Exhausted (&gt; 75% consumed)</h3>
<p><strong>Status</strong>: 🔴 Red</p>
<p><strong>Characteristics</strong>:</p>
<ul><li>More than 16.2 minutes consumed</li><li>SLO at risk of breach</li><li>Emergency response mode</li></ul>
<p><strong>Actions</strong>:</p>
<ul><li><strong>Development</strong>: 100% focus on reliability, zero feature work</li><li><strong>Releases</strong>: Complete feature freeze, emergency fixes only</li><li><strong>Changes</strong>: Incident response mode</li><li><strong>Monitoring</strong>: War room, twice-daily executive updates</li></ul>
<p><strong>Restrictions</strong>:</p>
<ul><li><strong>Complete feature freeze</strong>: No new features, period</li><li><strong>Emergency fixes only</strong>: Must be approved by VP Engineering or CTO</li><li><strong>Rigorous testing</strong>: All changes tested in staging with load testing</li><li><strong>Extended rollout</strong>: Manual, gradual rollout with constant monitoring</li></ul>
<p><strong>Required Actions</strong>:</p>
<ol><li><strong>Executive escalation</strong>: Notify VP Engineering, CTO immediately</li><li><strong>Root cause analysis</strong>: Identify all sources of budget consumption</li><li><strong>Remediation plan</strong>: Create detailed plan to restore reliability</li><li><strong>Daily reports</strong>: Written updates to executives twice daily</li><li><strong>Postmortem</strong>: Comprehensive review of what led to exhaustion</li></ol>
<p><strong>Meeting Cadence</strong>: Twice daily war room</p>
<p><strong>Approval Required</strong>: Yes (VP Engineering OR CTO)</p>
<p><strong>Communication</strong>:</p>
<ul><li>Slack: #incidents, #engineering-all, #executive</li><li>Email: [email protected], [email protected]</li><li>Status page: Public notification if customer-impacting</li></ul>
<hr>
<h2>Error Budget Attribution</h2>
<p>We track what consumes our error budget to identify improvement opportunities.</p>
<h3>Attribution Categories</h3>
<ol><li><strong>Infrastructure failures</strong> (cloud provider, network, hardware)</li><li><strong>Deployment-related</strong> (bad releases, config changes)</li><li><strong>Dependency failures</strong> (upstream services, databases)</li><li><strong>Traffic spikes</strong> (unexpected load, DDoS)</li><li><strong>Application bugs</strong> (code defects)</li><li><strong>Human error</strong> (manual operations, misconfigurations)</li><li><strong>Scheduled maintenance</strong> (planned downtime)</li></ol>
<h3>Monthly Attribution Report</h3>
<pre><code>Error Budget Consumption (October 2025)
Total consumed: 12.5 minutes (57.8%)
Attribution:
1. Deployment-related: 6.2 minutes (48.7%)
- Bad release (v2.5.1): 5 minutes
- Config rollout issue: 1.2 minutes
2. Dependency failures: 4.5 minutes (36%)
- Database failover: 3 minutes
- Auth service outage: 1.5 minutes
3. Infrastructure: 1.8 minutes (14.4%)
- AWS us-east-1 network issue: 1.8 minutes
Action Items:
- Improve pre-deployment testing (owner: Alice)
- Implement automatic rollback on error spike (owner: Bob)
- Add caching layer for auth service (owner: Charlie)</code></pre>
<h2>Burn Rate Alerting</h2>
<p>We use multi-window, multi-burn-rate alerts based on Google SRE best practices.</p>
<h3>Alert Thresholds</h3>
<p>| Window | Burn Rate | Impact | Alert | Action | |--------|-----------|--------|-------|--------| | 1 hour | 14.4x | 5% budget in 1 hour | Page | Immediate response | | 6 hours | 6x | 5% budget in 6 hours | Page | Urgent response | | 24 hours | 3x | 5% budget in 24 hours | Ticket | Schedule investigation |</p>
<h3>Example Burn Rates</h3>
<p><strong>Normal operation</strong>: 1x burn rate</p>
<ul><li>Consuming budget at expected rate</li><li>On track to use 100% by end of window</li></ul>
<p><strong>Fast burn</strong>: 14.4x burn rate</p>
<ul><li>Consuming budget 14.4x faster than normal</li><li>Will exhaust 5% of budget in 1 hour</li><li><strong>Action</strong>: Page on-call immediately, start incident response</li></ul>
<p><strong>Slow burn</strong>: 3x burn rate</p>
<ul><li>Consuming budget 3x faster than normal</li><li>Will exhaust budget in 10 days instead of 30</li><li><strong>Action</strong>: Create ticket, investigate during business hours</li></ul>
<h2>Budget Reset Policy</h2>
<h3>Rolling Window</h3>
<p>We use a 30-day rolling window, which means:</p>
<ul><li>Budget is continuously calculated</li><li>Old incidents &quot;age out&quot; after 30 days</li><li>No sudden resets (avoids cliff-edge behavior)</li></ul>
<p><strong>Implications</strong>:</p>
<ul><li>Past incidents impact budget for 30 days</li><li>Recent reliability improvements take time to reflect</li><li>More stable, predictable budget consumption</li></ul>
<h3>No Gaming</h3>
<p><strong>We do not reset budgets early</strong>. This would undermine the purpose of error budgets. If the budget is exhausted:</p>
<ol><li>We focus on reliability improvements</li><li>We defer feature work until budget recovers</li><li>We learn from what caused the exhaustion</li></ol>
<h2>Stakeholder Communication</h2>
<h3>Internal Communication</h3>
<p><strong>Zone 1-2</strong>: Standard Slack notifications <strong>Zone 3</strong>: Daily email updates to engineering <strong>Zone 4</strong>: Twice-daily updates to executives</p>
<h3>External Communication</h3>
<p><strong>Customer notification required when</strong>:</p>
<ul><li>SLA breach (&lt; 99.9% availability)</li><li>Major incidents lasting &gt; 15 minutes</li><li>Data integrity issues</li></ul>
<p><strong>Status page updates</strong>:</p>
<ul><li>All Sev1 incidents</li><li>Sev2 incidents lasting &gt; 30 minutes</li><li>Scheduled maintenance</li></ul>
<h2>Review and Adjustment</h2>
<h3>Quarterly SLO Review</h3>
<p><strong>Participants</strong>: SRE, Engineering, Product, Customer Success</p>
<p><strong>Agenda</strong>:</p>
<ol><li>Review past quarter achievement</li><li>Analyze error budget consumption patterns</li><li>Assess if SLO targets are appropriate</li><li>Update error budget policy if needed</li><li>Identify top reliability improvements</li></ol>
<p><strong>Questions to ask</strong>:</p>
<ul><li>Did we meet our SLO?</li><li>If yes, should we tighten it (too much budget left)?</li><li>If no, should we loosen it or improve reliability?</li><li>Is the error budget policy working?</li><li>Are teams responding appropriately to budget consumption?</li></ul>
<h3>Policy Adjustments</h3>
<p><strong>When to adjust thresholds</strong>:</p>
<ul><li>Team consistently exhausts budget: Loosen SLO or improve reliability</li><li>Team never uses budget: Tighten SLO to enable more velocity</li><li>Policy feels arbitrary: Adjust zones based on actual impact</li></ul>
<p><strong>Process</strong>:</p>
<ol><li>Propose changes in quarterly review</li><li>Get consensus from engineering and product</li><li>Document changes and rationale</li><li>Update this policy</li><li>Communicate to all stakeholders</li></ol>
<h2>Examples and Case Studies</h2>
<h3>Example 1: Healthy Budget Usage</h3>
<p><strong>Scenario</strong>: Early in month, 10% budget consumed</p>
<ul><li>Status: Zone 1 (Healthy)</li><li>Action: Continue normal operations</li><li>Velocity: Full speed ahead</li></ul>
<h3>Example 2: Bad Deployment</h3>
<p><strong>Scenario</strong>: Bad deployment consumes 30% budget in 2 hours</p>
<ul><li>Status: Zone 2 (Concerning)</li><li>Action:</li></ul>
<ol><li>Rollback deployment immediately</li><li>Mandatory postmortem</li><li>Implement better pre-deployment testing</li><li>Switch to Zone 2 restrictions</li><li>Monitor budget closely</li></ol>
<h3>Example 3: Approaching Exhaustion</h3>
<p><strong>Scenario</strong>: 70% budget consumed with 10 days left in window</p>
<ul><li>Status: Zone 3 (Critical)</li><li>Action:</li></ul>
<ol><li>Feature freeze on non-critical work</li><li>Daily standup focused on reliability</li><li>Identify and fix top sources of errors</li><li>Consider rolling back recent risky features</li><li>Notify VP Engineering</li><li>Create remediation plan</li></ol>
<h3>Example 4: Budget Exhausted</h3>
<p><strong>Scenario</strong>: 85% budget consumed with 15 days left</p>
<ul><li>Status: Zone 4 (Exhausted)</li><li>Action:</li></ul>
<ol><li>Complete feature freeze</li><li>Executive war room</li><li>Comprehensive root cause analysis</li><li>Detailed remediation plan</li><li>Twice-daily executive updates</li><li>Focus 100% on reliability until budget recovers</li></ol>
<h2>Appendix: Budget Calculation Formulas</h2>
<h3>Time-Based SLO</h3>
<pre><code>Error Budget (minutes) = Total Time × (1 - SLO Target)
Example (99.95% over 30 days):
Error Budget = 30 days × 24 hours × 60 minutes × (1 - 0.9995)
= 43,200 minutes × 0.0005
= 21.6 minutes</code></pre>
<h3>Request-Based SLO</h3>
<pre><code>Error Budget (requests) = Total Requests × (1 - SLO Target)
Example (99.9% success rate, 10M requests):
Error Budget = 10,000,000 × (1 - 0.999)
= 10,000 failed requests allowed</code></pre>
<h3>Burn Rate</h3>
<pre><code>Burn Rate = (Budget Consumed / Time Elapsed) / (1 / Window Duration)
Example (15% budget consumed in 3 days of 30-day window):
Expected Consumption = 3 / 30 = 10%
Actual Consumption = 15%
Burn Rate = 15% / 10% = 1.5x</code></pre>
<h3>Prediction</h3>
<pre><code>Hours Until Exhaustion = (Remaining Budget) / (Current Burn Rate)
Example (40% budget consumed in 120 hours):
Burn Rate = 0.4 / 120 = 0.00333 per hour
Remaining = 0.6
Hours Until Exhaustion = 0.6 / 0.00333 = 180 hours (7.5 days)</code></pre>
<hr>
<p><strong>Document Control</strong></p>
<p>| Version | Date | Author | Changes | |---------|------|--------|---------| | 2.0 | 2025-10-27 | SRE Team | Updated thresholds, added burn rate alerting | | 1.1 | 2025-08-15 | SRE Team | Added attribution tracking | | 1.0 | 2025-01-10 | SRE Team | Initial policy |</p>
<p><strong>Approval</strong>:</p>
<ul><li>Tech Lead: Alice Smith (2025-10-27)</li><li>Engineering Manager: Bob Jones (2025-10-27)</li><li>VP Engineering: Charlie Brown (2025-10-27)</li></ul>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/error-budget-policy.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>