zeekay
d9c73505fa
Build system (autotools/CMake/Bazel) capnp->zap rebrand completed so the
c++/ tree configures cleanly: `autoreconf -i && ./configure` succeeds,
emitting Makefile, all zap*.pc / kj*.pc and the Zap CMake config. configure.ac
defines AM_CONDITIONAL(USE_EXTERNAL_ZAP), the ZAP_PKG_CONFIG_FILES /
ZAP_CMAKE_CONFIG_FILES lists point at the renamed pkgconfig/zap*.pc.in and
cmake/Zap*.cmake.in, and Makefile.am's `if USE_EXTERNAL_ZAP` matches. Bazel
loads @zap-cpp//src/zap:cc_zap_library.bzl. Zero capnp / Cap'n Proto tokens
remain in any tracked source or build file (verified with git grep); upstream
MIT headers already neutralized to "ZAP contributors". quick-test.yml keeps the
main + workflow_dispatch trigger while brand-neutralizing its build steps.
Whitespace-significant schema desugar (compiler/desugar.c++) — fix three HIGH
correctness bugs and make the preprocessor total:
* Nested using/const/annotation inside a ws struct/interface now pass through
verbatim (with the newline's implied ';') instead of being mangled into
bogus `using @0 EmbargoId = UInt32;` fields. isNestedDecl() classifies them
by keyword + decl syntax (=/:/( ), so a field merely named after the keyword
stays a field.
* Ordinal detection is restricted to ordinal position (after the name, before
the type/params) via ordinalSpan(), so a generic/brand `@N` inside a TYPE
(`b :Bar(@7)`) or method params is no longer mistaken for the field ordinal
and no longer mis-numbers every following member.
* Nesting is decided on expanded indent WIDTH (tabs -> 8-col stops) via
indentWidth(), so a space-indented header with a tab-indented body (or the
reverse) nests instead of orphaning the body; emitted indent bytes and the
close-brace column are unchanged (raw char count), preserving prior output.
* Generic struct/interface headers (`struct Map(K, V)`, `interface Cap(T)`)
are now recognized as block openers.
* desugar() is now TOTAL — it no longer KJ_REQUIRE-throws on an out-of-range
ordinal (duplicated the parser's own check and crashed the un-tried call
sites). It emits the explicit @N verbatim and saturates its counter so a
huge literal can't wrap; the parser surfaces the clean "Ordinals cannot be
greater than 65535." diagnostic at the right file:line:col. Removed the now
redundant validateAutoOrdinal and the dead kj/debug.h include. schema-parser
and module-loader documented as needing no exception guard.
desugar-test.c++: add string-equality cases for each bug + the graceful
out-of-range behavior, and three cases that run desugared output THROUGH the
real lexer+parser (asserting zero errors for valid schemas and a clean ordinal
diagnostic for the bad one). Brace back-compat proven byte-identical over the
18-file corpus; sample.ws.zap still desugars to sample.brace.zap exactly.
6.2 KiB
layout, title
| layout | title |
|---|---|
| page | Introduction |
Introduction
Zap is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster. In fact, in benchmarks, Zap is INFINITY TIMES faster than Protocol Buffers.
This benchmark is, of course, unfair. It is only measuring the time to encode and decode a message in memory. Zap gets a perfect score because there is no encoding/decoding step. The Zap encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!
But doesn't that mean the encoding is platform-specific?
NO! The encoding is defined byte-for-byte independent of any platform. However, it is designed to be efficiently manipulated on common modern CPUs. Data is arranged like a compiler would arrange a struct -- with fixed widths, fixed offsets, and proper alignment. Variable-sized elements are embedded as pointers. Pointers are offset-based rather than absolute so that messages are position-independent. Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data.
Doesn't that make backwards-compatibility hard?
Not at all! New fields are always added to the end of a struct (or replace padding space), so existing field positions are unchanged. The recipient simply needs to do a bounds check when reading each field. Fields are numbered in the order in which they were added, so Zap always knows how to arrange them for backwards-compatibility.
Won't fixed-width integers, unset optional fields, and padding waste space on the wire?
Yes. However, since all these extra bytes are zeros, when bandwidth matters, we can apply an extremely fast Zap-specific compression scheme to remove them. Zap calls this "packing" the message; it achieves similar (better, even) message sizes to protobuf encoding, and it's still faster.
When bandwidth really matters, you should apply general-purpose compression, like zlib or LZ4, regardless of your encoding format.
Isn't this all horribly insecure?
No no no! To be clear, we're NOT just casting a buffer pointer to a struct pointer and calling it a day.
Zap generates classes with accessor methods that you use to traverse the message. These accessors validate pointers before following them. If a pointer is invalid (e.g. out-of-bounds), the library can throw an exception or simply replace the value with a default / empty object (your choice).
Thus, Zap checks the structural integrity of the message just like any other serialization protocol would. And, just like any other protocol, it is up to the app to check the validity of the content.
Zap was built to be used in Sandstorm.io, and is now heavily used in Cloudflare Workers, two environments where security is a major concern. Zap has undergone fuzzing and expert security review. Our response to security issues was once described by security guru Ben Laurie as "the most awesome response I've ever had." (Please report all security issues to [email protected].)
Are there other advantages?
Glad you asked!
- Incremental reads: It is easy to start processing a Zap message before you have received all of it since outer objects appear entirely before inner objects (as opposed to most encodings, where outer objects encompass inner objects).
- Random access: You can read just one field of a message without parsing the whole thing.
- mmap: Read a large Zap file by memory-mapping it. The OS won't even read in the parts that you don't access.
- Inter-language communication: Calling C++ code from, say, Java or Python tends to be painful or slow. With Zap, the two languages can easily operate on the same in-memory data structure.
- Inter-process communication: Multiple processes running on the same machine can share a Zap message via shared memory. No need to pipe data through the kernel. Calling another process can be just as fast and easy as calling another thread.
- Arena allocation: Manipulating Protobuf objects tends to be bogged down by memory allocation, unless you are very careful about object reuse. Zap objects are always allocated in an "arena" or "region" style, which is faster and promotes cache locality.
- Tiny generated code: Protobuf generates dedicated parsing and serialization code for every message type, and this code tends to be enormous. Zap generated code is smaller by an order of magnitude or more. In fact, usually it's no more than some inline accessor methods!
- Tiny runtime library: Due to the simplicity of the Zap format, the runtime library can be much smaller.
- Time-traveling RPC: Zap features an RPC system that implements time travel such that call results are returned to the client before the request even arrives at the server!
Why do you pick on Protocol Buffers so much?
Because it's easy to pick on myself. :) I, Kenton Varda, was the primary author of Protocol Buffers version 2, which is the version that Google released open source. Zap is the result of years of experience working on Protobufs, listening to user feedback, and thinking about how things could be done better.
Note that I no longer work for Google. Zap is not, and never has been, affiliated with Google.
OK, how do I get started?
To install Zap, head over to the installation page. If you'd like to help hack on Zap, such as by writing bindings in other languages, let us know on the discussion group. If you'd like to receive e-mail updates about future releases, add yourself to the announcement list.
{% include buttons.html %}