mirror of https://github.com/zap-proto/cpp-core.git synced 2026-10-30 15:06:00 +00:00

Files

T

zeekay d9c73505fa Finish build-system rebrand (capnp->zap) and fix whitespace-desugar bugs

Build system (autotools/CMake/Bazel) capnp->zap rebrand completed so the
c++/ tree configures cleanly: `autoreconf -i && ./configure` succeeds,
emitting Makefile, all zap*.pc / kj*.pc and the Zap CMake config. configure.ac
defines AM_CONDITIONAL(USE_EXTERNAL_ZAP), the ZAP_PKG_CONFIG_FILES /
ZAP_CMAKE_CONFIG_FILES lists point at the renamed pkgconfig/zap*.pc.in and
cmake/Zap*.cmake.in, and Makefile.am's `if USE_EXTERNAL_ZAP` matches. Bazel
loads @zap-cpp//src/zap:cc_zap_library.bzl. Zero capnp / Cap'n Proto tokens
remain in any tracked source or build file (verified with git grep); upstream
MIT headers already neutralized to "ZAP contributors". quick-test.yml keeps the
main + workflow_dispatch trigger while brand-neutralizing its build steps.

Whitespace-significant schema desugar (compiler/desugar.c++) — fix three HIGH
correctness bugs and make the preprocessor total:

  * Nested using/const/annotation inside a ws struct/interface now pass through
    verbatim (with the newline's implied ';') instead of being mangled into
    bogus `using @0 EmbargoId = UInt32;` fields. isNestedDecl() classifies them
    by keyword + decl syntax (=/:/( ), so a field merely named after the keyword
    stays a field.
  * Ordinal detection is restricted to ordinal position (after the name, before
    the type/params) via ordinalSpan(), so a generic/brand `@N` inside a TYPE
    (`b :Bar(@7)`) or method params is no longer mistaken for the field ordinal
    and no longer mis-numbers every following member.
  * Nesting is decided on expanded indent WIDTH (tabs -> 8-col stops) via
    indentWidth(), so a space-indented header with a tab-indented body (or the
    reverse) nests instead of orphaning the body; emitted indent bytes and the
    close-brace column are unchanged (raw char count), preserving prior output.
  * Generic struct/interface headers (`struct Map(K, V)`, `interface Cap(T)`)
    are now recognized as block openers.
  * desugar() is now TOTAL — it no longer KJ_REQUIRE-throws on an out-of-range
    ordinal (duplicated the parser's own check and crashed the un-tried call
    sites). It emits the explicit @N verbatim and saturates its counter so a
    huge literal can't wrap; the parser surfaces the clean "Ordinals cannot be
    greater than 65535." diagnostic at the right file:line:col. Removed the now
    redundant validateAutoOrdinal and the dead kj/debug.h include. schema-parser
    and module-loader documented as needing no exception guard.

desugar-test.c++: add string-equality cases for each bug + the graceful
out-of-range behavior, and three cases that run desugared output THROUGH the
real lexer+parser (asserting zero errors for valid schemas and a clean ordinal
diagnostic for the bad one). Brace back-compat proven byte-identical over the
18-file corpus; sample.ws.zap still desugars to sample.brace.zap exactly.

2026-06-20 00:02:20 -07:00

7.1 KiB

Raw Permalink Blame History

layout, title

layout	title
page	Road Map

Road Map

This is a list of big ideas we'd like to implement in Zap. We don't know in what order these will actually happen; as always, real work is driven by real-world needs.

Language Features

Inline lists: Lets you define a field composed of a fixed number of elements of the same type, and have those elements be stored directly within the struct rather than as a separate object. Useful mainly to avoid the need to validate list lengths when the length should always be the same. Also saves a pointer's worth of space.
Type aliases: Ability to define a type which is just an alias of some other type, and have it show up as e.g. a typedef in languages that support that. (The current using keyword is intended only for local use and does not affect code generation.)
Doc comments: Harvest doc comments from schema files and use them to generate doc comments on generated code. Also make them available in the compiled schema so that a documentation generator could use them.
Encapsulated types: This will allow you to create a hand-written wrapper around a type which will be automatically injected into the generated code, so that you can provide a nicer interface which encapsulates the type's inner state.
Maps: Based on encapsulated and parameterized types.

RPC Protocol Features

Dynamic schema transmission: Allow e.g. Python applications to obtain schemas directly from the RPC server so that they need not have a local copy. Great for interactive debugging.
Three-way introductions (level 3 RPC): Allow RPC interactions between more than two parties, with new connections formed automatically as needed.
Bulk and Realtime: Add features that make it easier to design Zap APIs for bulk data transfers (with flow control) and realtime communications (where it's better to drop messages than to deliver them late).
UDP transport: Zap RPC could benefit from implementing a UDP transport, in order to achieve zero-round-trip three-party introductions and to implement "realtime" APIs (see "bulk and realtime", above).
Encrypted transport: Zap RPC should support an encrypted transport which uses capability-based authorization (not PKI), can accomplish zero-round-trip three-party introductions (via a pre-shared key from the introducer) and based on modern crypto. TLS is not designed for this, but we don't want to invent new crypto; we intend to build on libsodium and the Noise Protocol Framework as much as possible.

C++ Zap API Features

Plain Old C Structs: The code generator should also generate a POCS type corresponding to each struct type. The POCS type would use traditional memory allocation, thus would not support zero-copy, but would support a more traditional and easy-to-use C++ API, including the ability to mutate the object over time without convoluted memory management. POCS types could be extracted from and inserted into messages with a single copy, allowing them to be used easily in non-performance-critical code.
Multi-threading: It should be made easy to assign different Zap RPC objects to different threads and have them be able to safely call each other. Each thread would still have an anyschronous event loop serving the objects that belong to it.
Shared memory RPC: Zero-copy inter-process communication.
JSON codec customization: Extend the JSON library to support customizing the JSON representation using annotations. For example, a field could be given a different name in JSON than it is in Zap. The goal of these features would be to allow any reasonable pre-existing JSON schema to be representable as a Zap type definition, so that servers implementing JSON APIs can use Zap exclusively on the server side.
LZ4 integration: Integrate LZ4 compression into the API to further reduce bandwidth needs with minimal CPU overhead.
Annotations API: For each annotation definition, generate code which assists in extracting that annotation from schema objects in a type-safe way.

C++ KJ API Features

KJ is a framework library that is bundled with Zap, but is broadly applicable to C++ applications even if they don't use Zap serialization.

Fiber-based concurrency: The C++ runtime's event loop concurrency model will be augmented with support for fibers, which are like threads except that context switches happen only at well-defined points (thus avoiding the need for mutex locking). Fibers essentially provide syntax sugar on top of the event loop model.
TLS bindings: Write bindings for e.g. OpenSSL to make it easy to integrate with the KJ I/O framework, Zap RPC, and the KJ HTTP library.
Modern crypto bindings: A thin wrapper around libsodium with a nice C++ API, e.g. representing keys using fixed-size, trivially-copyable classes.
Event loop integrations: We should provide off-the-shelf integrations with popular event loop libraries, such as libuv, libev, libevent, boost::asio, and others, so that it's easier to use Zap RPC in applications that already use another event framework.

Storage

ORM interface: Define a standard interface for capabilities that represent remotely-stored objects, with get, put, publish, and subscribe methods. Ideally, parameterize this interface on the stored type.
mmap-friendly mutable storage format: Define a standard storage format that is friendly to mmap-based use while allowing modification. (With the current serialization format, mmap is only useful for read-only structures.) Possibly based on the ORM interface, updates only possible at the granularity of a whole ORM entry.

Tools

Schema compatibility checker: Add a zap command which, given two schemas, verifies that the latter is a compatible upgrade from the former. This could be used as a git hook to prevent submission of schema changes that would break wire compatibility.
RPC debugger: Add a zap command which sends an RPC from the command line and prints the result. Useful for debugging RPC servers.

Quality Assurance

These things absolutely must happen before any 1.0 release. Note that it's not yet decided when a 1.0 release would happen nor how many 0.x releases might precede it.

Expand test coverage: There are lots of tests now, but some important scenarios, such as handling invalid of invalid input, need better testing.
Performance review: Performance is already very good compared to competitors, but at some point we need to break out the profiler and really hone down on the details.
Security review: We need a careful security review to make sure malicious input cannot crash an application or corrupt memory.

Infrastructure

Note: These are very large projects.

JSON-HTTP proxy: Develop a web server which can expose a Zap RPC backend as a JSON-over-HTTP protocol.
Database: A fast storage database based on Zap which implements the ORM interface on top of the mmap storage format.

7.1 KiB Raw Permalink Blame History