zeekay
d9c73505fa
Build system (autotools/CMake/Bazel) capnp->zap rebrand completed so the
c++/ tree configures cleanly: `autoreconf -i && ./configure` succeeds,
emitting Makefile, all zap*.pc / kj*.pc and the Zap CMake config. configure.ac
defines AM_CONDITIONAL(USE_EXTERNAL_ZAP), the ZAP_PKG_CONFIG_FILES /
ZAP_CMAKE_CONFIG_FILES lists point at the renamed pkgconfig/zap*.pc.in and
cmake/Zap*.cmake.in, and Makefile.am's `if USE_EXTERNAL_ZAP` matches. Bazel
loads @zap-cpp//src/zap:cc_zap_library.bzl. Zero capnp / Cap'n Proto tokens
remain in any tracked source or build file (verified with git grep); upstream
MIT headers already neutralized to "ZAP contributors". quick-test.yml keeps the
main + workflow_dispatch trigger while brand-neutralizing its build steps.
Whitespace-significant schema desugar (compiler/desugar.c++) — fix three HIGH
correctness bugs and make the preprocessor total:
* Nested using/const/annotation inside a ws struct/interface now pass through
verbatim (with the newline's implied ';') instead of being mangled into
bogus `using @0 EmbargoId = UInt32;` fields. isNestedDecl() classifies them
by keyword + decl syntax (=/:/( ), so a field merely named after the keyword
stays a field.
* Ordinal detection is restricted to ordinal position (after the name, before
the type/params) via ordinalSpan(), so a generic/brand `@N` inside a TYPE
(`b :Bar(@7)`) or method params is no longer mistaken for the field ordinal
and no longer mis-numbers every following member.
* Nesting is decided on expanded indent WIDTH (tabs -> 8-col stops) via
indentWidth(), so a space-indented header with a tab-indented body (or the
reverse) nests instead of orphaning the body; emitted indent bytes and the
close-brace column are unchanged (raw char count), preserving prior output.
* Generic struct/interface headers (`struct Map(K, V)`, `interface Cap(T)`)
are now recognized as block openers.
* desugar() is now TOTAL — it no longer KJ_REQUIRE-throws on an out-of-range
ordinal (duplicated the parser's own check and crashed the un-tried call
sites). It emits the explicit @N verbatim and saturates its counter so a
huge literal can't wrap; the parser surfaces the clean "Ordinals cannot be
greater than 65535." diagnostic at the right file:line:col. Removed the now
redundant validateAutoOrdinal and the dead kj/debug.h include. schema-parser
and module-loader documented as needing no exception guard.
desugar-test.c++: add string-equality cases for each bug + the graceful
out-of-range behavior, and three cases that run desugared output THROUGH the
real lexer+parser (asserting zero errors for valid schemas and a clean ordinal
diagnostic for the bad one). Brace back-compat proven byte-identical over the
18-file corpus; sample.ws.zap still desugars to sample.brace.zap exactly.
6.9 KiB
layout, title, author
| layout | title | author |
|---|---|---|
| post | Zap 0.6 Released: Two and a half years of improvements | kentonv |
Today we're releasing Zap 0.6, the first major Zap release in nearly 2.5 years.
Zap has been under active development the entire time, as part of its parent project, Sandstorm.io. The lack of releases did not indicate a lack of development, but rather a lack of keeping the code running on every platform it supports -- especially Windows. Without a working Windows build, we couldn't do a release. But as Sandstorm didn't need Windows, it was hard to prioritize -- that is, until contributors stepped up!
Note that this release encompasses the core tools and the C++ reference implementation. Implementations in other languages have their own release schedules, but it's likely that several will be updated soon to integrate new language features.
Brought to you by Cloudflare
As announced on the Sandstorm blog, most of the Sandstorm team (including myself) now work for Cloudflare. Cloudflare is one of the largest users of Zap, as described in this talk by John-Graham Cumming, and as such maintaining Zap is part of my job at Cloudflare.
What's New?
Full Windows / Visual Studio Support
With this release, all of Zap's functionality now works on Windows with Visual Studio 2015 and 2017. That includes the serialization, dynamic API, schema parser, async I/O framework (using I/O completion ports), RPC, and tools. This is a huge step up from 0.5, in which Zap could only be built in "lite mode", which supported only basic serialization.
Most of the work to make this happen was contributed by Harris Hancock (with some help from Gordon McShane, Mark Grimes, myself, and others). It was no small feat: Visual Studio's C++ compiler is still quite buggy, so lots of work-arounds were needed. Meanwhile, the Zap developers working on Linux were continuously introducing new issues with their changes. Harris sorted it all out and delivered a beautiful series of patches. He also helped get us set up with continuous integration on AppVeyor, so that we can stay on top of these issues going forward.
Security Hardening
The 0.6 release includes a number of measures designed to harden Zap's C++ implementation against possible security bugs. These include:
- The core pointer validation code has been refactored to detect possible integer overflows at compile time using C++ template metaprogramming, as described in this old blog post.
- The core test suite -- which runs when you type
make check-- now includes a targeted fuzz test of the pointer validation code. - We additionally tested this release using American Fuzzy Lop, running several different test cases for over three days each.
JSON converter
Zap messages can now be converted to and from JSON using libzap-json. This makes it easy to integrate your JSON front-end API with your Zap back-end.
See the zap/compat/json.h header for API details.
This library was primarily built by Kamal Marhubi and Branislav Katreniak, using Zap's dynamic API.
HTTP library
KJ (the C++ framework library bundled with Zap) now ships with a minimalist HTTP library, libkj-http. The library is based on the KJ asynchronous I/O framework and covers both client-side and server-side use cases. Although functional and used in production today, the library should be considered a work in progress -- expect improvements in future releases, such as client connection pooling and TLS support.
See the kj/compat/http.h header for API details.
Smaller things
With two years of development, there are far too many changes to list, but here are some more things:
- KJ now offers its own unit test framework under
kj/test.h, as well as a compatibility shim with Google Test underkj/compat/gtest.h. The KJ and Zap tests no longer depend on Google Test. - New API
zap::TextCodecinzap/serialize-text.hprovides direct access to parse text-format Zap messages (requireslibzapc, the schema parser library). (Contributed by: Philip Quinn) - It is possible to compare Zap messages for equality (with correct handling of unknown fields, something Protocol Buffers struggled with) using APIs in
zap/any.h. (Contributed by: Joshua Warner) - A function
zap::canonicalize()has been added which returns the canonical serialization of a given struct. (Contributed by: Matthew Maurer) AnyPointerfields can now be assigned in constant values, by referencing another named constant (which itself is defined with a specific type).- In addition to
AnyPointer, the typesAnyStruct,AnyList, andCapabilitycan now be used in schemas. - New class
zap::CapabilityServerSetinzap/capability.hallows an RPC server to detect when capabilities to its own local objects are passed back to it and allows it to "unwrap" them to get at the underlying native object. - A membrane framework library was added (header
zap/membrane.h). This makes it easy to set up a MITM layer between RPC actors, e.g. to implement revocability, transformations, and many other useful capability patterns. - Basic flow control can now be applied to an RPC connection, preventing new messages from being accepted if outstanding calls exceed a certain watermark, which helps prevent excessive buffering / malicious resource exhaustion. See
RpcSystem::setFlowLimit(). - KJ's networking API now includes datagram protocols (UDP).
- In
.zapsyntax, all comma-delimited lists can now have a trailing comma. (Contributed by: Drew Fisher) - Hundreds more small feature additions and bug fixes.