dev/content/docs/benchmarks.mdx

---
title: Benchmarks
description: Honest performance numbers from the zap-proto/bench harness.
---

# Benchmarks

Every number on this page is reproducible from [zap-proto/bench](https://github.com/zap-proto/bench). If a claim isn't traceable to a benchmark file checked into that repo, it doesn't live here. We've burned trust with fabricated tables before; this page exists as an antidote.

## What's measured today

The current harness (`bench_test.go`) measures **HTTP/1.1+JSON vs ZAP-binary** across four workload sizes: tiny (16 B body, 4 headers), small (256 B, 8 headers), medium (4 KiB, 12 headers), large (64 KiB, 8 headers). Source: [bench_test.go](https://github.com/zap-proto/bench/blob/main/bench_test.go). Raw output: [bench-results.txt](https://github.com/zap-proto/bench/blob/main/bench-results.txt).

### Single-client throughput

5000 sequential requests, single TCP connection, keep-alive.

| Workload | HTTP/1.1+JSON | ZAP-binary | ratio |
|---|---|---|---|
| tiny (16 B) | 12,202 req/s | 11,232 req/s | 0.92× |
| small (256 B) | 8,822 req/s | 9,892 req/s | **1.12×** |
| medium (4 KiB) | 7,487 req/s | 7,582 req/s | 1.01× |
| large (64 KiB) | 3,875 req/s | 1,920 req/s | 0.50× |

### Concurrent throughput (32 in-flight)

32 concurrent workers × 1000 requests each.

| Workload | HTTP/1.1+JSON | ZAP-binary | ratio |
|---|---|---|---|
| small | 34,411 req/s | 30,868 req/s | 0.90× |
| medium | 15,503 req/s | 13,907 req/s | 0.90× |

### Wire bytes

How big is the request on the wire?

| Workload | HTTP bytes | ZAP bytes | ratio |
|---|---|---|---|
| tiny | 222 | 196 | **0.88×** (ZAP smaller) |
| small | 478 | 444 | 0.93× |
| medium | 4,322 | 4,388 | 1.02× |
| large | 65,844 | 66,196 | 1.01× |

## What the numbers say

The honest read:

- **At small payloads ZAP is mildly faster** and uses ~10% fewer wire bytes than HTTP+JSON. Wins are modest.
- **At medium payloads it's a tie.** Both are dominated by socket and OS overhead.
- **At large payloads (64 KiB) ZAP loses by 2×** — the current ZAP-HTTP path has higher allocations and an extra copy in the hot loop. Optimization work is queued; results will land here when re-measured.
- **Under concurrency 32, ZAP loses 10%** uniformly. The hot path doesn't yet take advantage of arena pooling that the encoder is capable of.

The hand-wavy "10× faster than gRPC" / "1438× faster consensus" tables that lived here before were not measured. They were aspirational claims dressed as data. Removed.

## Where ZAP actually wins today

Performance is not the strongest argument for ZAP in 2026. The stronger arguments — supported by mechanism, not by µs — are:

1. **One wire format, many protocols.** [HTTP](/docs/protocols/http), [MCP](/docs/protocols/mcp), [A2A](/docs/protocols/a2a), [FIX](/docs/protocols/fix) all ride the same framing. One observability story, one auth story, one cipher suite to audit.
2. **PQ at the transport.** [IETF X-Wing](/docs/post-quantum) is the default. Browsers got `X25519MLKEM768` in 2025; ZAP gives you the same KEM directly inside your application protocol, not just terminated at the edge.
3. **Capabilities, not tokens.** [The capability mailbox example](/docs/examples/capability-mailbox) shows what's structurally impossible to express in HTTP: a reference that the holder can call but cannot forge, transferable to a third peer, revocable without rotating keys.
4. **PQ-RNS.** [Name resolution that doesn't trust a CA.](/docs/protocols/rns) The KEM key in the resolved Record is the key the client dials — no separate cert chain.

These are the levers. When the harness produces wins on µs/req, they'll land here, sourced from the run.

## Run it yourself

### Go harness (HTTP-shaped workloads)

```bash
git clone https://github.com/zap-proto/bench
cd bench
go test -v -run Test 2>&1 | tee my-bench-results.txt
```

Tweak the workloads in `bench_test.go`. Hardware will move the numbers; the **ratios** should be reproducible across machines.

### Rust harness (carsales / catrank / eval)

The canonical workload suite — three workloads (carsales, catrank, eval), three modes (object / bytes / pipe), two compressions (none / packed), two scratch policies (reuse / no-reuse). 20 distinct measurements per run.

The benchmark uses only the **`zap` crate** (the renamed runtime) and **`zapc` crate** (the renamed codegen). Schema files are `.zap`; pre-compiled `*_zap.rs` is committed in-tree so `cargo build -p benchmark` invokes **zero external schema tools**. `cargo tree -p benchmark` shows no `capnp` dependency anywhere.

```bash
git clone https://github.com/zap-proto/rust
cd rust
bash ./benchmark/bench.sh quick     # ~10 sec sanity
bash ./benchmark/bench.sh default   # original iteration counts, ~1 min
bash ./benchmark/bench.sh full      # 10× iterations, ~10 min
```

Each run writes one JSON file to `bench-results/$(hostname)-$(timestamp).json`. The schema is documented in [zap-proto/rust/benchmark/README.md](https://github.com/zap-proto/rust/blob/main/benchmark/README.md):

```json
{
  "schema_version": 1,
  "host": {"hostname":"...","os":"...","arch":"..."},
  "toolchain": {"rustc":"..."},
  "iters": {"carsales":10000,"catrank":1000,"eval":200000},
  "runs": [
    {"case":"carsales","mode":"object","compression":"none",
     "scratch":"reuse","iters":10000,"elapsed_secs":0.42,
     "throughput_ops_per_sec":23809.5},
    ...
  ]
}
```

Twenty runs per file (carsales × 2 scratch × 5 modes/compressions + catrank × 5 + eval × 5).

### Contribute results across hardware

The whole point of producing structured JSON is so we can collect numbers across hardware classes — server CPUs vs laptops vs embedded vs cloud VMs. Open a PR against [zap-proto/rust](https://github.com/zap-proto/rust/tree/main/benchmark) dropping your `bench-results/*.json` into the `bench-results/` directory. The schema is stable; future aggregation will pull from whatever's in there.

A sample run from this dev machine (Apple M-series, macOS aarch64, `quick` preset, `eval` case only):

| mode | compression | iters | elapsed | ops/s |
|---|---|---|---|---|
| object | none | 10000 | 0.025s | 405,986 |
| bytes | none | 10000 | 0.029s | 339,869 |
| bytes | packed | 10000 | 0.045s | 223,006 |
| pipe | none | 10000 | 0.123s | 81,007 |
| pipe | packed | 10000 | 0.201s | 49,760 |

Object-mode wins because there's no encode/decode step — the in-memory layout *is* the wire layout. Pipe is slowest because it crosses a process boundary. Packed adds compute for compression in exchange for fewer bytes on the wire. The shape of these numbers should hold across machines; absolute throughput will swing with CPU and cache size.

### Parallel saturation

`benchmark/saturate.sh` runs N worker copies of the encoder workload in parallel and computes **real** parallel efficiency against a recorded single-worker baseline. Honest measurement, not aggregate-divided-by-per-core (which is tautological).

```bash
# 1. record the single-worker baseline
PRESET=full bash ./benchmark/saturate.sh baseline

# 2. saturate with N=ncpu workers (default), get real speedup
PRESET=full bash ./benchmark/saturate.sh

# 3. eyeball CPU usage during the run
tail bench-results/saturate-*/cpu.log
```

Sample numbers on the dev machine (Apple M1 Max, 10 cores: 8 P + 2 E):

```
single-worker      : 6.003s for 2,000,000 iters = 333,183 ops/s
10-worker wall     : 11.46s
aggregate ops/s    : 1,745,201
REAL speedup       : 5.24× (= single × N / wall)
effective cores    : 5.24 of 10
parallel efficiency: 52%
```

52% efficiency on a 10-core M1 Max is normal: macOS QoS scheduler defaults subprocesses to efficiency cores, the system allocator serializes across processes, and unified memory bandwidth is shared. The CPU sampler captures the actual peak (~557% in `ps -o %cpu`, ≈5.5 cores busy), which matches the speedup math.

If you submit a `bench-results/` JSON from a different machine, the matching `saturate-*/cpu.log` is welcome too — it's the proof-of-life that the workers actually ran in parallel.

## Methodology

- Single host, loopback (`127.0.0.1`)
- Both HTTP and ZAP use bufio readers/writers sized 64 KiB
- TCP `NoDelay = true` on both
- Persistent keep-alive connections (no per-request TCP establishment)
- 5000-request warm window, allocs/GC measured via runtime/metrics
- Both servers are minimal echo handlers — what you're measuring is framing overhead, not application logic

This is *not* a comprehensive perf suite. It's a single, honest harness in-tree. Native-ZAP-vs-gRPC, MCP-bridge-vs-stdio, FIX-vs-FIX-over-TCP — each needs its own harness, each is roadmap work, each will be added to this page when the harness file exists in [zap-proto/bench](https://github.com/zap-proto/bench).