Files
skills/public/protocols-protobuf-schemas.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

693 lines
35 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>protocols-protobuf-schemas</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/protocols-protobuf-schemas.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/protocols-protobuf-schemas">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/protocols-protobuf-schemas.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>protocols-protobuf-schemas</h1><p class="lede">Protocol Buffers schema design, evolution, and code generation</p><h1>Protocol Buffers Schemas</h1>
<p><strong>Scope</strong>: Proto syntax, schema design, field numbering, backward/forward compatibility, code generation, schema registry <strong>Lines</strong>: ~350 <strong>Last Updated</strong>: 2025-10-27</p>
<h2>When to Use This Skill</h2>
<p>Activate this skill when:</p>
<ul><li>Designing Protocol Buffer schemas from scratch</li><li>Implementing schema evolution strategies</li><li>Managing schema versioning and compatibility</li><li>Generating code for multiple languages</li><li>Integrating with gRPC or Kafka</li><li>Setting up schema registries</li><li>Validating schema best practices</li><li>Migrating between proto2 and proto3</li></ul>
<h2>Core Concepts</h2>
<h3>What are Protocol Buffers?</h3>
<p><strong>Protocol Buffers</strong> (protobuf): Language-neutral, platform-neutral mechanism for serializing structured data.</p>
<p><strong>Key characteristics</strong>:</p>
<ul><li><strong>Compact binary format</strong>: 3-10x smaller than XML/JSON</li><li><strong>Fast</strong>: 20-100x faster than XML/JSON</li><li><strong>Type-safe</strong>: Strong typing enforced by schema</li><li><strong>Forward/backward compatible</strong>: Schema evolution without breaking clients</li><li><strong>Multi-language</strong>: Code generation for 10+ languages</li><li><strong>Self-describing</strong>: Schema embedded in generated code</li></ul>
<p><strong>Use cases</strong>:</p>
<ul><li>gRPC service definitions</li><li>Kafka message serialization</li><li>Configuration files</li><li>Data storage formats</li><li>Inter-service communication</li></ul>
<h3>Proto3 vs Proto2</h3>
<p><strong>Proto3</strong> (recommended):</p>
<pre><code class="lang-protobuf">syntax = &quot;proto3&quot;; // Always specify
message User {
string id = 1;
string name = 2;
int32 age = 3; // Default value: 0
repeated string tags = 4; // Default: empty list
}</code></pre>
<p><strong>Proto2</strong> (legacy):</p>
<pre><code class="lang-protobuf">syntax = &quot;proto2&quot;;
message User {
required string id = 1; // Must be set
optional string name = 2; // May be set
optional int32 age = 3 [default = 0];
repeated string tags = 4;
}</code></pre>
<p><strong>Key differences</strong>:</p>
<ul><li>Proto3: No <code>required</code>/<code>optional</code> keywords (all fields optional)</li><li>Proto3: No default values (uses type defaults)</li><li>Proto3: Simpler syntax, better performance</li><li>Proto2: Explicit presence detection</li></ul>
<hr>
<h2>Schema Design Basics</h2>
<h3>Field Numbers</h3>
<p><strong>Critical rule</strong>: Field numbers are permanent identifiers used in binary encoding.</p>
<pre><code class="lang-protobuf">message User {
string id = 1; // Field number 1
string name = 2; // Field number 2
int32 age = 3; // Field number 3
}</code></pre>
<p><strong>Best practices</strong>:</p>
<ul><li><strong>1-15</strong>: Single-byte encoding (use for frequent fields)</li><li><strong>16-2047</strong>: Two-byte encoding (use for less frequent fields)</li><li><strong>19000-19999</strong>: Reserved by Protocol Buffers</li><li><strong>Never reuse</strong> deleted field numbers (use <code>reserved</code>)</li></ul>
<h3>Reserved Fields</h3>
<p>Prevent field number/name reuse:</p>
<pre><code class="lang-protobuf">message User {
reserved 5, 8 to 10; // Reserved numbers
reserved &quot;old_field&quot;, &quot;deprecated_field&quot;; // Reserved names
string id = 1;
string name = 2;
// Field 5 cannot be reused
}</code></pre>
<h3>Data Types</h3>
<p><strong>Scalar types</strong>:</p>
<pre><code class="lang-protobuf">message Example {
// Integers
int32 age = 1; // -2^31 to 2^31-1
int64 user_id = 2; // -2^63 to 2^63-1
uint32 count = 3; // 0 to 2^32-1
sint32 delta = 4; // Signed (efficient for negatives)
// Floating point
float price = 5;
double precise = 6;
// Boolean
bool is_active = 7;
// String (UTF-8 or ASCII)
string name = 8;
// Bytes (arbitrary byte sequence)
bytes data = 9;
}</code></pre>
<p><strong>Well-known types</strong> (import from <code>google/protobuf/*</code>):</p>
<pre><code class="lang-protobuf">import &quot;google/protobuf/timestamp.proto&quot;;
import &quot;google/protobuf/duration.proto&quot;;
import &quot;google/protobuf/any.proto&quot;;
message Event {
google.protobuf.Timestamp created_at = 1;
google.protobuf.Duration timeout = 2;
google.protobuf.Any metadata = 3; // Can hold any message type
}</code></pre>
<h3>Repeated Fields (Lists)</h3>
<pre><code class="lang-protobuf">message User {
repeated string tags = 1; // List of strings
repeated Address addresses = 2; // List of nested messages
}
// Packed encoding (more efficient for numeric types)
message Metrics {
repeated int32 values = 1 [packed = true]; // Proto3 default
}</code></pre>
<h3>Enums</h3>
<pre><code class="lang-protobuf">enum Status {
STATUS_UNSPECIFIED = 0; // Always have 0 as first value
STATUS_ACTIVE = 1;
STATUS_INACTIVE = 2;
STATUS_PENDING = 3;
}
message User {
Status status = 1;
}</code></pre>
<p><strong>Best practices</strong>:</p>
<ul><li>First value must be 0 (default in proto3)</li><li>Use <code>_UNSPECIFIED</code> suffix for zero value</li><li>Prefix enum values with enum name to avoid conflicts</li></ul>
<h3>Nested Messages</h3>
<pre><code class="lang-protobuf">message User {
string id = 1;
Address address = 2;
// Nested message (scoped to User)
message Address {
string street = 1;
string city = 2;
string postal_code = 3;
}
}
// Usage: User.Address</code></pre>
<h3>Oneof (Union Types)</h3>
<p>Only one field can be set:</p>
<pre><code class="lang-protobuf">message SearchRequest {
oneof query {
string text_query = 1;
int32 id_query = 2;
bool all_query = 3;
}
}
// Set text_query → id_query and all_query are cleared</code></pre>
<h3>Maps</h3>
<pre><code class="lang-protobuf">message User {
map&lt;string, string&gt; attributes = 1; // Key must be scalar (except float/double/bytes)
map&lt;int32, Address&gt; addresses = 2;
}</code></pre>
<hr>
<h2>Schema Evolution</h2>
<h3>Backward Compatibility</h3>
<p><strong>Old clients</strong> can read <strong>new data</strong>.</p>
<p><strong>Safe changes</strong>:</p>
<ul><li>✅ Add new fields (old clients ignore them)</li><li>✅ Delete fields (mark as <code>reserved</code>)</li><li>✅ Add new enum values (old clients see as unknown)</li></ul>
<p><strong>Breaking changes</strong>:</p>
<ul><li>❌ Change field number</li><li>❌ Change field type (string ↔ int32, etc.)</li><li>❌ Rename fields (wire format uses numbers, but affects JSON)</li><li>❌ Change repeated ↔ singular</li><li>❌ Reuse deleted field numbers</li></ul>
<p><strong>Example</strong>:</p>
<pre><code class="lang-protobuf">// V1
message User {
string id = 1;
string name = 2;
}
// V2 (backward compatible)
message User {
string id = 1;
string name = 2;
string email = 3; // New field (old clients ignore)
reserved 4; // Deleted field
repeated string tags = 5;
}</code></pre>
<h3>Forward Compatibility</h3>
<p><strong>New clients</strong> can read <strong>old data</strong>.</p>
<p><strong>Safe changes</strong>:</p>
<ul><li>✅ Add new fields with defaults</li><li>✅ Delete fields (new client handles missing data)</li></ul>
<p><strong>Breaking changes</strong>:</p>
<ul><li>❌ Make field required (proto2)</li><li>❌ Remove default values (proto2)</li></ul>
<h3>Deprecation</h3>
<pre><code class="lang-protobuf">message User {
string id = 1;
string name = 2 [deprecated = true]; // Mark as deprecated
string full_name = 3; // Replacement field
}</code></pre>
<hr>
<h2>Code Generation</h2>
<h3>Multi-Language Support</h3>
<p><strong>Python</strong>:</p>
<pre><code class="lang-bash"># Install compiler
pip install grpcio-tools
# Generate code
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. user.proto
# Output: user_pb2.py, user_pb2_grpc.py</code></pre>
<p><strong>Go</strong>:</p>
<pre><code class="lang-bash"># Install plugins
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
# Generate code
protoc --go_out=. --go_opt=paths=source_relative \
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
user.proto
# Output: user.pb.go, user_grpc.pb.go</code></pre>
<p><strong>Java</strong>:</p>
<pre><code class="lang-bash"># Add plugin to pom.xml
protoc --java_out=src/main/java user.proto
# Output: User.java</code></pre>
<p><strong>TypeScript</strong>:</p>
<pre><code class="lang-bash"># Install plugins
npm install -D protoc-gen-ts protoc-gen-grpc-web
# Generate code
protoc --ts_out=. --grpc-web_out=import_style=typescript,mode=grpcwebtext:. user.proto
# Output: user_pb.ts</code></pre>
<h3>Buf (Modern Build Tool)</h3>
<p><strong>buf.yaml</strong>:</p>
<pre><code class="lang-yaml">version: v1
breaking:
use:
- FILE
lint:
use:
- DEFAULT</code></pre>
<p><strong>buf.gen.yaml</strong> (code generation):</p>
<pre><code class="lang-yaml">version: v1
plugins:
- name: go
out: gen/go
opt: paths=source_relative
- name: go-grpc
out: gen/go
opt: paths=source_relative
- name: python
out: gen/python
- name: python-grpc
out: gen/python</code></pre>
<p><strong>Usage</strong>:</p>
<pre><code class="lang-bash"># Install
brew install bufbuild/buf/buf
# Lint schemas
buf lint
# Generate code
buf generate
# Check breaking changes
buf breaking --against .git#branch=main</code></pre>
<hr>
<h2>Schema Registry Integration</h2>
<h3>Confluent Schema Registry (Kafka)</h3>
<pre><code class="lang-python">from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.protobuf import ProtobufSerializer
# Schema Registry client
schema_registry_client = SchemaRegistryClient({
&#x27;url&#x27;: &#x27;http://localhost:8081&#x27;
})
# Protobuf serializer
protobuf_serializer = ProtobufSerializer(
User, # Generated class
schema_registry_client,
{&#x27;use.deprecated.format&#x27;: False}
)
# Producer
producer = SerializingProducer({
&#x27;bootstrap.servers&#x27;: &#x27;localhost:9092&#x27;,
&#x27;value.serializer&#x27;: protobuf_serializer
})
# Send message
user = User(id=&#x27;123&#x27;, name=&#x27;Alice&#x27;)
producer.produce(&#x27;users&#x27;, value=user)
producer.flush()</code></pre>
<h3>Pulsar Schema Registry</h3>
<pre><code class="lang-python">import pulsar
client = pulsar.Client(&#x27;pulsar://localhost:6650&#x27;)
# Producer with Protobuf schema
producer = client.create_producer(
&#x27;users&#x27;,
schema=pulsar.schema.ProtobufSchema(User)
)
user = User(id=&#x27;123&#x27;, name=&#x27;Alice&#x27;)
producer.send(user)</code></pre>
<hr>
<h2>Common Patterns</h2>
<h3>Request/Response Pattern (gRPC)</h3>
<pre><code class="lang-protobuf">service UserService {
rpc GetUser(GetUserRequest) returns (GetUserResponse);
rpc ListUsers(ListUsersRequest) returns (ListUsersResponse);
}
message GetUserRequest {
string id = 1;
}
message GetUserResponse {
User user = 1;
string error = 2; // Error message if failed
}
message ListUsersRequest {
int32 page_size = 1;
string page_token = 2;
}
message ListUsersResponse {
repeated User users = 1;
string next_page_token = 2;
}</code></pre>
<h3>Domain Modeling</h3>
<pre><code class="lang-protobuf">// Event sourcing
message UserCreatedEvent {
string user_id = 1;
string email = 2;
google.protobuf.Timestamp created_at = 3;
}
message UserUpdatedEvent {
string user_id = 1;
map&lt;string, string&gt; updated_fields = 2;
google.protobuf.Timestamp updated_at = 3;
}
// Aggregate root
message User {
string id = 1;
string email = 2;
UserProfile profile = 3;
repeated Address addresses = 4;
google.protobuf.Timestamp created_at = 5;
google.protobuf.Timestamp updated_at = 6;
}</code></pre>
<hr>
<h2>Anti-Patterns</h2>
<p>❌ <strong>Reusing field numbers</strong>: Breaks backward compatibility ✅ Use <code>reserved</code> for deleted fields</p>
<p>❌ <strong>Changing field types</strong>: <code>string</code> → <code>int32</code> breaks wire format ✅ Add new field with new type, deprecate old field</p>
<p>❌ <strong>Using field numbers &gt; 2047 for frequent fields</strong>: Inefficient encoding ✅ Use 1-15 for hot fields, 16-2047 for cold fields</p>
<p>❌ <strong>Not using <code>reserved</code></strong>: Developers might reuse numbers/names ✅ Always mark deleted fields as reserved</p>
<p>❌ <strong>Enum without zero value</strong>: Default is 0, must be defined ✅ Always have <code>NAME_UNSPECIFIED = 0</code> for enums</p>
<p>❌ <strong>Large messages (&gt; 1MB)</strong>: Performance issues ✅ Split into smaller messages or use streaming</p>
<p>❌ <strong>No schema versioning</strong>: Can&#x27;t track changes ✅ Use package versioning (<code>users.v1</code>, <code>users.v2</code>)</p>
<hr>
<h2>Level 3: Resources</h2>
<h3>Overview</h3>
<p>This skill includes comprehensive Level 3 resources for deep Protocol Buffers schema design knowledge and practical automation tools.</p>
<p><strong>Resources include</strong>:</p>
<ul><li><strong>REFERENCE.md</strong> (3,200+ lines): Complete technical reference covering all protobuf concepts</li><li><strong>3 executable scripts</strong>: Schema validation, code generation, compatibility analysis</li><li><strong>9 production examples</strong>: Complete schema designs, evolution examples, multi-language integration</li></ul>
<h3>REFERENCE.md</h3>
<p><strong>Location</strong>: <code>skills/protocols/protobuf-schemas/resources/REFERENCE.md</code></p>
<p><strong>Comprehensive technical reference</strong> (3,200+ lines) covering:</p>
<p><strong>Core Topics</strong>:</p>
<ul><li>Protocol Buffers fundamentals and architecture</li><li>Proto2 vs Proto3 syntax and migration</li><li>Field types (scalar, repeated, maps, oneofs)</li><li>Well-known types (Timestamp, Duration, Any, etc.)</li><li>Schema evolution (backward/forward compatibility)</li><li>Field numbering and reserved fields</li><li>Code generation for Python, Go, Java, TypeScript, C++</li><li>Schema registry integration (Confluent, Pulsar)</li><li>Performance characteristics and optimization</li><li>JSON mapping and custom options</li><li>Advanced patterns (event sourcing, CQRS, domain modeling)</li></ul>
<p><strong>Key Sections</strong>:</p>
<ol><li><strong>Fundamentals</strong>: Binary encoding, wire format, serialization</li><li><strong>Syntax</strong>: Proto3 syntax, field types, modifiers</li><li><strong>Schema Design</strong>: Best practices, naming conventions, versioning</li><li><strong>Evolution</strong>: Backward/forward compatibility rules, migration strategies</li><li><strong>Code Generation</strong>: Multi-language support, custom options, plugins</li><li><strong>Well-Known Types</strong>: Timestamp, Duration, Any, Empty, FieldMask</li><li><strong>Schema Registry</strong>: Confluent, Pulsar, versioning strategies</li><li><strong>Performance</strong>: Encoding efficiency, message size, serialization speed</li><li><strong>Advanced Patterns</strong>: Event sourcing, domain modeling, API design</li><li><strong>Tools</strong>: protoc, buf, grpcurl, schema validators</li><li><strong>Anti-Patterns</strong>: Common mistakes and solutions</li></ol>
<p><strong>Format</strong>: Markdown with extensive code examples in multiple languages</p>
<h3>Scripts</h3>
<p>Three production-ready executable scripts in <code>resources/scripts/</code>:</p>
<h4>1. validate_proto_schemas.py (680 lines)</h4>
<p><strong>Purpose</strong>: Parse and validate Protocol Buffer schemas</p>
<p><strong>Features</strong>:</p>
<ul><li>Parse .proto files (proto2 and proto3)</li><li>Validate syntax and field numbering</li><li>Check naming conventions (PascalCase, snake_case)</li><li>Detect breaking changes between versions</li><li>Verify reserved fields and field number ranges</li><li>Validate imports and dependencies</li><li>Check best practices (enum zero values, field number efficiency)</li><li>Output as JSON or human-readable text</li></ul>
<p><strong>Usage</strong>:</p>
<pre><code class="lang-bash"># Basic validation
./validate_proto_schemas.py --proto-file user.proto
# JSON output
./validate_proto_schemas.py --proto-file user.proto --json
# Check breaking changes
./validate_proto_schemas.py --proto-file user_v2.proto --check-breaking --baseline user_v1.proto
# Validate directory
./validate_proto_schemas.py --proto-dir ./protos --json</code></pre>
<p><strong>Categories checked</strong>:</p>
<ul><li>Syntax errors and parsing issues</li><li>Field number conflicts and ranges</li><li>Naming convention violations</li><li>Reserved field usage</li><li>Import validation</li><li>Best practice recommendations</li></ul>
<h4>2. generate_proto_code.py (750 lines)</h4>
<p><strong>Purpose</strong>: Generate code for multiple languages from proto schemas</p>
<p><strong>Features</strong>:</p>
<ul><li>Support Python, Go, Java, TypeScript, C++</li><li>Custom options and plugins</li><li>Validate generated code structure</li><li>Handle dependencies and imports</li><li>Support buf.yaml and buf.gen.yaml</li><li>Generate example usage code</li><li>JSON output for automation</li></ul>
<p><strong>Usage</strong>:</p>
<pre><code class="lang-bash"># Generate Python code
./generate_proto_code.py --proto-file user.proto --language python --output-dir ./gen/python
# Generate multiple languages
./generate_proto_code.py --proto-file user.proto --languages python,go,java --output-dir ./gen
# Use buf for generation
./generate_proto_code.py --proto-dir ./protos --use-buf --output-dir ./gen
# JSON output
./generate_proto_code.py --proto-file user.proto --language go --json</code></pre>
<p><strong>Supported languages</strong>:</p>
<ul><li>Python: <code><em>_pb2.py</code>, <code></em>_pb2_grpc.py</code></li><li>Go: <code><em>.pb.go</code>, <code></em>_grpc.pb.go</code></li><li>Java: <code>*.java</code></li><li>TypeScript: <code><em>_pb.ts</code>, <code></em>_pb_service.ts</code></li><li>C++: <code><em>.pb.h</code>, <code></em>.pb.cc</code></li></ul>
<h4>3. analyze_schema_compatibility.py (650 lines)</h4>
<p><strong>Purpose</strong>: Compare schema versions and detect breaking changes</p>
<p><strong>Features</strong>:</p>
<ul><li>Compare two schema versions</li><li>Detect breaking vs non-breaking changes</li><li>Verify backward/forward compatibility</li><li>Suggest migration paths</li><li>Validate evolution rules</li><li>Generate compatibility report</li><li>JSON output for CI/CD</li></ul>
<p><strong>Usage</strong>:</p>
<pre><code class="lang-bash"># Compare schemas
./analyze_schema_compatibility.py --old user_v1.proto --new user_v2.proto
# Check backward compatibility
./analyze_schema_compatibility.py --old user_v1.proto --new user_v2.proto --mode backward
# JSON output for automation
./analyze_schema_compatibility.py --old user_v1.proto --new user_v2.proto --json
# Detailed migration report
./analyze_schema_compatibility.py --old user_v1.proto --new user_v2.proto --migration-guide</code></pre>
<p><strong>Detects</strong>:</p>
<ul><li>Field number changes (breaking)</li><li>Field type changes (breaking)</li><li>Field deletions (potentially breaking)</li><li>Field additions (safe)</li><li>Enum value changes</li><li>Message nesting changes</li><li>Service changes (gRPC)</li></ul>
<h3>Examples</h3>
<p>Nine production-ready examples in <code>resources/examples/</code>:</p>
<h4>1. user_service.proto</h4>
<p>Complete service definition with CRUD operations, nested messages, enums, and well-known types.</p>
<h4>2. schema_evolution/</h4>
<p>Directory with v1, v2, v3 of a schema demonstrating evolution:</p>
<ul><li>Adding fields (backward compatible)</li><li>Deprecating fields</li><li>Using reserved fields</li><li>Type changes (safe patterns)</li></ul>
<h4>3. breaking_vs_nonbreaking.md</h4>
<p>Comprehensive guide with examples of breaking vs non-breaking changes.</p>
<h4>4. code_generation/</h4>
<p>Multi-language code generation examples:</p>
<ul><li>Python client/server</li><li>Go client/server</li><li>Java implementation</li><li>TypeScript integration</li></ul>
<h4>5. schema_registry/</h4>
<p>Integration examples:</p>
<ul><li>Confluent Schema Registry (Kafka)</li><li>Pulsar Schema Registry</li><li>Schema versioning strategies</li></ul>
<h4>6. grpc_service/</h4>
<p>Complete gRPC service definition with:</p>
<ul><li>Unary, server streaming, client streaming, bidirectional RPCs</li><li>Error handling patterns</li><li>Request/response wrappers</li></ul>
<h4>7. kafka_messages/</h4>
<p>Kafka message schemas with:</p>
<ul><li>Event sourcing patterns</li><li>CQRS command/query separation</li><li>Protobuf serialization</li></ul>
<h4>8. buf_configuration/</h4>
<p>buf.yaml and buf.gen.yaml examples:</p>
<ul><li>Linting rules</li><li>Breaking change detection</li><li>Multi-language code generation</li></ul>
<h4>9. ci_validation/</h4>
<p>CI/CD pipeline examples:</p>
<ul><li>GitHub Actions workflow</li><li>GitLab CI pipeline</li><li>Schema validation in CI</li></ul>
<h3>Quick Start</h3>
<p><strong>1. Validate schemas</strong>:</p>
<pre><code class="lang-bash">cd skills/protocols/protobuf-schemas/resources/scripts
./validate_proto_schemas.py --proto-file ../examples/user_service.proto --json</code></pre>
<p><strong>2. Generate code</strong>:</p>
<pre><code class="lang-bash">./generate_proto_code.py --proto-file ../examples/user_service.proto --languages python,go --output-dir ./gen</code></pre>
<p><strong>3. Check compatibility</strong>:</p>
<pre><code class="lang-bash">./analyze_schema_compatibility.py --old ../examples/schema_evolution/v1.proto --new ../examples/schema_evolution/v2.proto</code></pre>
<h3>File Structure</h3>
<pre><code>skills/protocols/protobuf-schemas/
├── protobuf-schemas.md (this file)
└── resources/
├── REFERENCE.md (3,200+ lines)
├── scripts/
│ ├── validate_proto_schemas.py (680 lines)
│ ├── generate_proto_code.py (750 lines)
│ └── analyze_schema_compatibility.py (650 lines)
└── examples/
├── user_service.proto
├── schema_evolution/
│ ├── v1.proto
│ ├── v2.proto
│ └── v3.proto
├── breaking_vs_nonbreaking.md
├── code_generation/
├── schema_registry/
├── grpc_service/
├── kafka_messages/
├── buf_configuration/
└── ci_validation/</code></pre>
<h3>Resources Summary</h3>
<p>| Category | Item | Lines | Description | |----------|------|-------|-------------| | <strong>Reference</strong> | REFERENCE.md | 3,200+ | Complete technical reference | | <strong>Scripts</strong> | validate_proto_schemas.py | 680 | Schema validator | | | generate_proto_code.py | 750 | Code generator | | | analyze_schema_compatibility.py | 650 | Compatibility analyzer | | <strong>Examples</strong> | 9 examples | 2,000+ | Production-ready schemas |</p>
<p><strong>Total</strong>: 7,000+ lines of production-ready resources</p>
<hr>
<p><strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/protocols-protobuf-schemas.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>