hanzoai/base
2
0
Fork 0
mirror of https://github.com/hanzoai/base.git synced 2026-10-21 03:41:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Document how to run, extend and upgrade Base

Browse Source 
The README quickstart, docs/versions.md, hooks.md, rules.md, config.md
and listeners.md, and examples/hooks-hybrid-search, each command run
against the binary built from examples/base. LLM.md is cut to a current
map with test commands and known problems; threat-model, NETWORK,
SECURITY and CONTRIBUTING are corrected to the code.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-09-11 10:52:05 -07:00
co-authored by Hanzo Dev
parent 9a96f79bad
commit 2ceb430a12
16 changed files with 1257 additions and 1719 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+35 -68
View File
@@ -1,82 +1,49 @@
# Contributing to Base
# Contributing
Thanks for taking the time to improve Base!
Base is one Go module. `examples/base` is the binary, and everything else is
packages it imports.
This document describes how to prepare a PR for a change in the main repository.
## Go
- [Prerequisites](#prerequisites)
- [Making changes in the Go code](#making-changes-in-the-go-code)
- [Making changes in the Admin UI](#making-changes-in-the-admin-ui)
You need the Go version in `go.mod` (1.26.8) or newer.
## Prerequisites
- Go 1.25+ (for making changes in the Go code)
- Node 18+ (for making changes in the Admin UI)
If you haven't already, you can fork the main repository and clone your fork so that you can work locally:
```
git clone https://github.com/your_username/base.git
```sh
cd examples/base
go run . serve --http 127.0.0.1:8090
```
> [!IMPORTANT]
> It is recommended to create a new branch from master for each of your bugfixes and features.
> This is required if you are planning to submit multiple PRs in order to keep the changes separate for review until they eventually get merged.
Check your change the way CI does, with the settings the published binary is
built with:
## Making changes in the Go code
```sh
GOWORK=off CGO_ENABLED=0 GOEXPERIMENT=jsonv2 go vet ./...
GOWORK=off CGO_ENABLED=0 GOEXPERIMENT=jsonv2 go test -count=1 ./...
golangci-lint run -c ./golangci.yml ./...
```
Base is distributed as a Go package, which means that in order to run the project you'll have to create a Go `main` program that imports the package.
On macOS two tests fail for want of RAM-backed storage: `TestTasksEmbed` in
`core` and `TestOrgBaseIsEncryptedAtRest` in `plugins/org`. The encrypted SQLite
codec will only decrypt into RAM, which is `/dev/shm` on Linux, or on macOS a
`tmpfs` mount named by `HANZO_SQLITE_RAMFS_DIR`.
The repository already includes such program, located in `examples/base`, that is also used for the prebuilt executables.
## Admin UI
So, let's assume that you already done some changes in the Base Go code and you want now to run them:
The admin is a React app in `ui-react/`, built with pnpm. Its build output,
`ui-react/dist`, is committed and embedded in the binary, so building Go needs
no Node toolchain.
1. Navigate to `examples/base`
2. Run `go run main.go serve`
```sh
pnpm --dir ui-react install
pnpm --dir ui-react dev # http://localhost:3000, sends /v1 to localhost:8090
pnpm --dir ui-react build
pnpm --dir ui-react smoke # loads the built admin in Playwright and checks what renders
```
This will start a web server on `http://localhost:8090` with the embedded prebuilt Admin UI from `ui/dist`. And that's it!
Commit a rebuilt `dist/` with any change under `ui-react/src`, and build it on
Linux: a macOS build of the same source emits a different, larger bundle, and
the next Linux build undoes it.
**Before making a PR to the main repository, it is a good idea to:**
## Pull requests
- Add unit/integration tests for your changes (we are using the standard `testing` go package).
To run the tests, you could execute (while in the root project directory):
```sh
go test ./...
# or using the Makefile
make test
```
- Run the linter - **golangci** ([see how to install](https://golangci-lint.run/usage/install/#local-installation)):
```sh
golangci-lint run -c ./golangci.yml ./...
# or using the Makefile
make lint
```
## Making changes in the Admin UI
Base Admin UI is a single-page application (SPA) built with Svelte and Vite.
To start the Admin UI:
1. Navigate to the `ui` project directory
2. Run `npm install` to install the node dependencies
3. Start vite's dev server
```sh
npm run dev
```
You could open the browser and access the running Admin UI at `http://localhost:3000`.
Since the Admin UI is just a client-side application, you need to have the Base backend server also running in the background (either manually running the `examples/base/main.go` or download a prebuilt executable).
> [!NOTE]
> By default, the Admin UI is expecting the backend server to be started at `http://localhost:8090`, but you could change that by creating a new `ui/.env.development.local` file with `HZ_BACKEND_URL = YOUR_ADDRESS` variable inside it.
Every change you make in the Admin UI should be automatically reflected in the browser at `http://localhost:3000` without reloading the page.
Once you are done with your changes, you have to build the Admin UI with `npm run build`, so that it can be embedded in the go package. And that's it - you can make your PR to the main Base repository.
Open an issue before a large change. Keep a pull request to one change, with
its tests, and update `docs/` when behaviour a reader relies on changes.
LLM.md
+153 -1314
View File
File diff suppressed because it is too large Load Diff
README.md
+82 -294
View File
@@ -2,337 +2,125 @@
# Base
The embedded, **SQLite-first** application backend for the Hanzo cloud — per-tenant data, in-process extension runtimes, and Hanzo IAM built in. One Go binary; the storage substrate every multi-tenant Hanzo Go service builds on.
Base is a backend in one Go binary: collections of records behind a REST API at
`/v1`, with realtime updates, files, per-collection access rules and JavaScript
hooks, stored in SQLite. Identity comes from Hanzo IAM, and with IAM on, each
org's data lives in a database file of its own.
[![Status](https://img.shields.io/badge/status-beta-blue)]()
[![License](https://img.shields.io/badge/license-MIT-blue)]()
## Run it
## Quick start
```bash
go install github.com/hanzoai/base/examples/base@latest
base serve
```
Then open the admin dashboard at `http://127.0.0.1:8090/_/`.
> [!NOTE]
> Base is under active development. Full backward compatibility is not guaranteed before `v1.0.0`.
## What this is
[Base](https://hanzo.ai/base) is an open source Go backend that gives every Hanzo Go service the same multi-tenant substrate. It includes:
- embedded **in-memory, SQL, and vector** data with **realtime subscriptions**
- built-in **files and users management**
- a convenient **Admin dashboard UI**
- **GraphQL** and a REST-style **`/v1` API**
- native **analytics and AI observability**
- **Hanzo IAM** as the one and only auth path (OIDC / PKCE)
- deep integration with the **[Hanzo AI Cloud](https://hanzo.ai)** for scale on day one
Each org gets its own per-tenant data file with a per-org KMS-derived DEK; `replicate` streams the WAL to age-encrypted object storage. Per-record validators, computed fields, and access rules run inside the same process via in-process extension runtimes (goja, wazero, pyvm, starkvm).
Use **[Hanzo App](https://hanzo.app)** to rapidly iterate and build new apps.
## API clients
The easiest way to talk to the Base web API is one of the official SDK clients:
- **JavaScript — [@hanzoai/js-sdk](https://github.com/hanzoai/js-sdk)** (_Browser, Node.js, React Native_)
- **Dart — [@hanzoai/dart-sdk](https://github.com/hanzoai/dart-sdk)** (_Web, Mobile, Desktop, CLI_)
See the full guide at [docs.hanzo.ai](https://docs.hanzo.ai).
## Overview
### Use as a standalone app
Download the prebuilt executable for your platform from the [Releases page](https://github.com/hanzoai/base/releases), extract the archive, and run `./base serve`.
The prebuilt executables are built from [`examples/base/main.go`](https://github.com/hanzoai/base/blob/main/examples/base/main.go) and ship with a JavaScript plugin enabled by default, so you can extend Base with JavaScript (see [Extend with JavaScript](https://docs.hanzo.ai/js-overview/)).
### Use as a Go framework / toolkit
Base is distributed as a regular Go library, so you can build your own app-specific business logic and still ship a single portable executable.
Minimal example:
0. [Install Go 1.26+](https://go.dev/doc/install) (_if you haven't already_).
1. Create a new project directory with the following `main.go`:
```go
package main
import (
"log"
"github.com/hanzoai/base"
"github.com/hanzoai/base/core"
)
func main() {
app := base.New()
app.OnServe().BindFunc(func(se *core.ServeEvent) error {
// registers a new "GET /hello" route
se.Router.GET("/hello", func(re *core.RequestEvent) error {
return re.String(200, "Hello world!")
})
return se.Next()
})
if err := app.Start(); err != nil {
log.Fatal(err)
}
}
```
2. Initialize dependencies: `go mod init myapp && go mod tidy`.
3. Start the app: `go run main.go serve`.
4. Build a statically linked executable: `CGO_ENABLED=0 go build`, then start it with `./myapp serve`.
See [Extend with Go](https://docs.hanzo.ai/go-overview/) for details.
### Building the repo example
To build the minimal standalone executable (like the prebuilt releases), run `go build` inside `examples/base`:
0. [Install Go 1.26+](https://go.dev/doc/install) (_if you haven't already_).
1. Clone the repo.
2. Navigate to `examples/base`.
3. Run `GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build` (see the [Go environment reference](https://go.dev/doc/install/source#environment)).
4. Start the executable: `./base serve`.
The pure-Go SQLite driver currently supports these build targets:
```
darwin amd64 linux arm64 linux s390x
darwin arm64 linux loong64 windows 386
freebsd amd64 linux ppc64le windows amd64
freebsd arm64 linux riscv64 windows arm64
linux 386 linux arm
linux amd64
```
### Testing
Base comes with a mixed bag of unit and integration tests. Run them with the standard `go test` command:
You need Go 1.26.8 or newer.
```sh
go test ./...
git clone --depth 1 https://github.com/hanzoai/base
cd base/examples/base
CGO_ENABLED=0 go build
```
See the [Testing guide](https://docs.hanzo.ai/testing) to learn how to write your own application tests.
Describe a collection in a migration. Base applies migrations when it starts.
## SQLite replication
Base opens SQLite through the encrypted `hanzoai/sqlite` driver (per-principal CEK). The `hanzoai/replicate` sidecar streams WAL changes to object storage, encrypted end-to-end with `luxfi/age`.
K8s sidecar pattern:
- **Init container** — restores the latest snapshot from object storage on startup.
- **Sidecar** — continuously replicates the WAL while the service runs.
Configure `age-identities` / `age-recipients` for end-to-end encryption of replicated data.
## Specs
Base implements:
- **HIP-0105** — In-Process Extension Runtime Standard
- **HIP-0107** — Streaming Replication over VFS
- **HIP-0302** — Encrypted SQLite + ZapDB Durability
- **HIP-0111** — Hanzo IAM (the one auth path)
It is the storage backend for every multi-tenant subsystem in **HIP-0106**.
## Architecture
```
http request -> zip.App -> base.App
|
per-tenant data/{orgSlug}.db (SQLite or ZapDB)
|
KMS-derived DEK replicate -> S3 (age-encrypted)
|
in-process extension runtime (HIP-0105):
goja (JS) | wazero (WASM) | pyvm (Python) | starkvm (Starlark)
```sh
mkdir -p base_/migrations
cat > base_/migrations/1_notes.js <<'EOF'
migrate((app) => {
app.save(new Collection({
type: "base",
name: "notes",
listRule: "",
viewRule: "",
createRule: "",
fields: [{ name: "text", type: "text", required: true }],
}))
})
EOF
./base serve --http 127.0.0.1:8090
```
## CLI
The `base cli` subcommand is a complete HTTP client for operating any running Base-backed daemon from the command line. It works against `base` or any binary that embeds Base.
### Targeting a server
```bash
# Defaults to http://127.0.0.1:8090 if nothing is set
base cli --url http://localhost:8090 collection list
# Or use environment variables
export BASE_URL=http://localhost:8090
export BASE_TOKEN=eyJhbGciOi...
base cli collection list
```
Server started at http://127.0.0.1:8090
├─ REST API: http://127.0.0.1:8090/v1/
└─ Dashboard: http://127.0.0.1:8090/
```
Global flags (apply to all subcommands):
From a second terminal, create a record and list the collection:
| Flag | Env var | Default | Description |
|------|---------|---------|-------------|
| `--url` | `BASE_URL` | `http://127.0.0.1:8090` | Server URL |
| `--token` | `BASE_TOKEN` | `~/.config/base/token` | Auth token |
| `--tenant` | — | — | Sets `X-Org-Id` header |
| `--format` | — | `table` on TTY, `json` otherwise | `table`, `json`, or `yaml` |
```sh
curl -s -X POST http://127.0.0.1:8090/v1/collections/notes/records \
-H 'Content-Type: application/json' -d '{"text":"hello"}'
{"collectionId":"hbc_3395098727","collectionName":"notes","id":"kbx358lmic2a4bw","text":"hello"}
### Authentication
```bash
# Login as a regular user
base cli login --email [email protected] --password secret123
# Login as superuser
base cli login --email [email protected] --password secret123 --superuser
# Check who you are
base cli whoami
curl -s http://127.0.0.1:8090/v1/collections/notes/records
{"items":[{"collectionId":"hbc_3395098727","collectionName":"notes","id":"kbx358lmic2a4bw","text":"hello"}],"page":1,"perPage":30,"totalItems":1,"totalPages":1}
```
The token is stored at `~/.config/base/token` (or `$XDG_CONFIG_HOME/base/token`) with mode `0600`.
The empty rules let anyone read and write `notes`; [docs/rules.md](docs/rules.md)
shows how to close them. The data is in `base_/data`.
### Collections
The startup output has two stale lines. The Dashboard address answers only when
`BASE_ENABLE_ADMIN_UI=1` is set, and the hint to run `superuser upsert` names a
command that was removed. Superusers come from IAM.
```bash
# List all collections
base cli collection list
## Use it as a Go library
# Show a specific collection's schema
base cli collection get users
# Export schema as JSON (always JSON, ignores --format)
base cli collection schema users > schema.json
```
### Records
```bash
# List records with filtering and sorting
base cli record list posts --filter "title~'hello'" --limit 10 --sort "-created"
# Get a single record
base cli record get posts abc123
# Create a record
base cli record create posts '{"title":"Hello","body":"World"}'
# Update a record
base cli record update posts abc123 '{"title":"Updated"}'
# Delete a record
base cli record delete posts abc123
```
### Crons
```bash
# List registered cron schedules
base cli crons list
```
### Using from a downstream daemon
Any Base-backed daemon can expose these subcommands without duplicating code. For example, in a downstream `main.go`:
In an empty directory, save this as `main.go`:
```go
package main
import (
"log"
"log"
"github.com/hanzoai/base"
"github.com/hanzoai/base/cmd"
"github.com/hanzoai/base"
"github.com/hanzoai/base/core"
)
func main() {
app := base.New()
app := base.New()
// ... register domain-specific hooks and routes ...
app.OnServe().BindFunc(func(se *core.ServeEvent) error {
se.Router.GET("/hello", func(re *core.RequestEvent) error {
return re.String(200, "Hello")
})
return se.Next()
})
// Register system commands for a flattened CLI:
// `myapp collection list` instead of `myapp cli collection list`
app.RootCmd.AddCommand(cmd.NewSuperuserCommand(app))
app.RootCmd.AddCommand(cmd.NewServeCommand(app, true))
cmd.AddCLISubcommands(app.RootCmd)
if err := app.Execute(); err != nil {
log.Fatal(err)
}
if err := app.Start(); err != nil {
log.Fatal(err)
}
}
```
Then operate the running daemon:
```bash
myapp collection list
myapp record list posts --filter "title~'hello'" --limit 20
myapp crons list
myapp daemon status
```sh
go mod init example.com/hello
go mod tidy
CGO_ENABLED=0 go build
./hello serve
```
### Registration patterns
`GET http://127.0.0.1:8090/hello` then answers `Hello`, next to the whole `/v1`
API.
**Flattened** (recommended for domain daemons) — commands at the root level:
## Read next
```go
app.RootCmd.AddCommand(cmd.NewSuperuserCommand(app))
app.RootCmd.AddCommand(cmd.NewServeCommand(app, true))
cmd.AddCLISubcommands(app.RootCmd) // collection, record, login, whoami, crons, daemon
app.Execute()
```
| Page | For |
|---|---|
| [docs/versions.md](docs/versions.md) | which tags serve what, and moving from `/api` and passwords to `/v1` and IAM |
| [docs/rules.md](docs/rules.md) | access rules, identity from IAM, orgs |
| [docs/hooks.md](docs/hooks.md) | JavaScript hooks and migrations |
| [docs/config.md](docs/config.md) | every command, flag and environment variable |
| [docs/listeners.md](docs/listeners.md) | every socket Base can open |
| [docs/threat-model.md](docs/threat-model.md) | what Base defends, and what you configure |
| [examples/hooks-hybrid-search](examples/hooks-hybrid-search) | full-text and vector search written as hooks |
**Nested** (default via `app.Start()`) — commands under a `cli` parent:
## Clients
```go
app.Start() // registers serve, superuser, cli (with all subcommands)
// Access via: myapp cli collection list
```
- JavaScript and TypeScript: [`@hanzo/base`](https://github.com/hanzo-js/base).
Give it the origin, as in `new BaseClient('http://127.0.0.1:8090')`; it adds
`/v1` itself.
- Dart: [hanzo-dart/base](https://github.com/hanzo-dart/base).
### Daemon lifecycle
## Contributing and security
The `daemon` subcommand manages the process lifecycle:
```bash
myapp daemon start # local: nohup spawn
myapp daemon stop # local: kill
myapp daemon status # local: pgrep
myapp daemon logs --follow # local: tail -f
myapp daemon restart # local: stop + start
myapp daemon status --env dev # K8s: kubectl get pods
myapp daemon restart --env test --yes # K8s: rollout restart
```
K8s actions require `--env` (`dev`, `test`, `main`) and are dry-run by default. Pass `--yes` to execute.
## Security
Report security vulnerabilities to **[email protected]**. All reports are addressed promptly, and you'll be credited in the fix release notes.
## Contributing
Base is free and open source under the [MIT License](LICENSE.md).
- [Open an issue](https://github.com/hanzoai/base/issues) for bugs and feature requests.
- Read [CONTRIBUTING.md](CONTRIBUTING.md) before sending a PR.
- For new features, open an issue to discuss the design first.
Bug fixes, optimizations, documentation improvements, and new OAuth2 providers are always welcome.
## Hanzo — the Open AI Cloud
Open source · every language · on-chain settlement. [hanzo.ai](https://hanzo.ai) · [docs.hanzo.ai](https://docs.hanzo.ai)
**SDKs in every language** — [Python](https://github.com/hanzoai/python-sdk) (flagship) · [TypeScript](https://github.com/hanzo-js/sdk) · [Go](https://github.com/hanzo-go/sdk) · [Rust](https://github.com/hanzo-rs/sdk) · [C++](https://github.com/hanzo-cpp/sdk) · [Swift](https://github.com/hanzo-swift/sdk) · [Kotlin](https://github.com/hanzo-kt/sdk) · [umbrella](https://github.com/hanzoai/sdk)
[CONTRIBUTING.md](CONTRIBUTING.md) covers building and testing. Report
vulnerabilities as [SECURITY.md](SECURITY.md) describes. Base is MIT licensed;
see [LICENSE.md](LICENSE.md).
SECURITY.md
+10 -5
View File
@@ -2,16 +2,21 @@
## Reporting a vulnerability
Email security@hanzo.ai with details. Encrypt with our PGP key (fingerprint TBD).
Email security@hanzo.ai with details.
We respond within 48 hours. Critical issues receive same-day acknowledgment.
## Scope
This policy covers code in this repository. For the broader Hanzo platform threat model, see [hanzoai/HIPs](https://github.com/hanzoai/HIPs).
This policy covers code in this repository. What Base defends on its own, and
what it leaves to a deployment, is in [docs/threat-model.md](docs/threat-model.md).
## Sandbox boundary
## Isolation
`base` enforces tenant isolation at the storage layer: each org gets its own per-tenant data file (`data/{orgSlug}.db`) with a per-org HKDF-derived DEK from KMS, and the WAL is shipped to age-encrypted object storage via `hanzoai/replicate`. User-supplied per-record validators, computed fields, and access rules execute exclusively inside the HIP-0105 in-process runtimes (goja, wazero, pyvm, starkvm) — never in the host Go process directly.
With `IAM_ENDPOINT` set, each org's data is its own SQLite file,
`<data dir>/orgs/<org>/data.db`, so a query in one org cannot read another's
rows. The `base` binary opens those files unencrypted: encryption needs a master
key from KMS, which the org plugin reads only when it is given an `IAMOrg`.
For runtime sandbox guarantees, see HIP-0105 (in-process extension runtimes).
JavaScript hooks run inside the Base process with its full authority. They are
not sandboxed.
docs/NETWORK.md
+26 -29
View File
@@ -1,6 +1,6 @@
# Base Network
Seamless, PQ-native, consensus-replicated HA for every Base app.
PQ-native, consensus-replicated HA for every Base app.
One mechanism. One env flag. Replica count `N ∈ {1,2,3,…}` works
identically on a laptop (`docker compose`) and on k8s (via
`~/work/hanzo/operator`). No FUSE, no LiteFS, no Consul.
@@ -8,7 +8,7 @@ identically on a laptop (`docker compose`) and on k8s (via
Durability and replication come from `luxfi/consensus/protocol/quasar`
— the same DAG consensus that runs Lux validators — applied to SQLite
WAL frames, one DAG per shard. Archive tier is optional cold storage
on `~/work/hanzo/storage` (`hanzoai/s3`) or GCS for PITR.
in S3 (`s3://`) for PITR.
## Problem
@@ -159,12 +159,8 @@ per-shard WAL segments to object storage. Layout:
of PQ-signed, quasar-finalized frames. Replay any prefix ⇒ SQLite at
that `txseq`. PITR for free.
Backend is pluggable via `github.com/hanzoai/s3` client surface:
- S3 (`s3://…`) — the S3 API, served by `hanzoai/s3`.
- GCS (`gs://…`) — native client, used for production in GCP-hosted
clusters.
- off — archive disabled; durability = quasar DAG only.
The backend is S3 (`s3://bucket/prefix`), served by `hanzoai/s3` or AWS.
`off` disables the archive, leaving the quasar DAG as the only durability.
### Env surface (one shape, every Base app)
@@ -175,11 +171,12 @@ Backend is pluggable via `github.com/hanzoai/s3` client surface:
| `BASE_REPLICATION` | `1` \| `2` \| `3` \| … | ≤ `BASE_PEERS` count. |
| `BASE_PEERS` | CSV of `host:port` DNS | operator-emitted in k8s; explicit in compose. |
| `BASE_NODE_ROLE` | `validator` (default) \| `archive` | |
| `BASE_ARCHIVE` | `gs://…` \| `s3://…` \| `off` (default) | |
| `BASE_LISTEN_HTTP` | `:8090` default | Base HTTP. |
| `BASE_ARCHIVE` | `s3://bucket/prefix` \| `off` (default) | |
| `BASE_LISTEN_HTTP` | `:8090` default | Read, but opens nothing; `serve --http` sets the HTTP address. |
| `BASE_LISTEN_P2P` | `host:port`, `:9999` default | quasar p2p listener. The host is honoured: `:9999` is every interface, `127.0.0.1:9999` loopback only. |
| `BASE_SHARD_BACKLOG_MAX` | bytes (64 MiB default) | R6 per-shard archive backlog cap; drop-oldest beyond. |
| `BASE_SHARD_BACKLOG_SEGMENTS` | integer (100 000 default) | R6 segment-count cap; first-to-hit with MAX drops. |
The R6 backlog caps are constants in `network/archive.go`: 64 MiB and 100,000
segments per shard. No variable changes them.
**The peer transport presents no certificate.**
@@ -452,23 +449,23 @@ never diverges. Same code path as prod.
## Acceptance tests
1. **Single-node**: `N=1`, `archive=off` → behaves exactly like
today. No regression in `core/base_test.go`.
2. **Docker compose N=3**: write 1000 rows, kill `base-b`, read all
1000 from `base-a` and `base-c`. Bring `base-b` back, verify it
catches up (txseq matches). No row loss, no duplication.
3. **Archive PITR**: write 1000 rows, record txseq, write 100 more.
Restore to the first txseq from GCS archive; DB has exactly 1000.
4. **k8s operator**: apply a Base-backed CR with
`network.replication: 3`. Verify: headless svc, 3 pods with
BASE_PEERS env, HPA, PDB minAvailable=2, archive sidecar when
`archive != off`.
5. **Autoscale**: drive `base_hot_shards` > `hotShardsTarget` via
synthetic load; verify KEDA scales Deployment up, new pod joins
the network, existing shards rebalance.
6. **Gateway**: two users in different shards produce writes on
different owner pods; both succeed; read-your-writes respected
via `txseq` cookie.
- **Single-node**: `N=1`, `archive=off` → behaves exactly like
today. No regression in `core/base_test.go`.
- **Docker compose N=3**: write 1000 rows, kill `base-b`, read all
1000 from `base-a` and `base-c`. Bring `base-b` back, verify it
catches up (txseq matches). No row loss, no duplication.
- **Archive PITR**: write 1000 rows, record txseq, write 100 more.
Restore to the first txseq from the S3 archive; DB has exactly 1000.
- **k8s operator**: apply a Base-backed CR with
`network.replication: 3`. Verify: headless svc, 3 pods with
BASE_PEERS env, HPA, PDB minAvailable=2, archive sidecar when
`archive != off`.
- **Autoscale**: drive `base_hot_shards` > `hotShardsTarget` via
synthetic load; verify KEDA scales Deployment up, new pod joins
the network, existing shards rebalance.
- **Gateway**: two users in different shards produce writes on
different owner pods; both succeed; read-your-writes respected
via `txseq` cookie.
## Non-goals
docs/config.md
+195
View File
@@ -0,0 +1,195 @@
# Configuration
Every command, flag and environment variable the `base` binary reads, taken
from the code. A Go program that embeds Base gets the root flags from
`base.New`, the `serve` flags, the core variables, and the variables of the
plugins it registers.
## Commands
| Command | What it does |
|---|---|
| `base serve [domain...]` | Runs the server. Naming domains turns on HTTPS with certificates from Let's Encrypt. |
| `base migrate up\|down [n]\|create <name>\|collections\|history-sync` | Applies, reverts or writes migrations. See [hooks](hooks.md#migrations). |
| `base update` | Replaces the binary with the latest GitHub release. No current release carries binaries, so it stops with `missing asset`. |
| `base cli <command>` | An HTTP client for a running Base: `collection`, `record`, `whoami`, `login`, `crons`, `daemon`, `status`, `rpc`, `config`, `self`, `cluster`, `operator`. |
## Root flags
These apply to every command.
| Flag | Default | Meaning |
|---|---|---|
| `--dir` | `$DATA_DIR`, else `base_/data` beside the path the binary was started by | Data directory. `./base` uses `./base_/data`; a `base` found on `PATH` uses `base_/data` in the working directory. |
| `--dev` | `false`, or `true` under `go run` | Log SQL statements and debug output. |
| `--encryptionEnv` | none | Name of an environment variable whose 32-character value encrypts stored settings. |
| `--queryTimeout` | `30` | Seconds before a `SELECT` is cancelled. |
| `--zap` | `$ZAP_ADDR`, else the HTTP host on `$ZAP_PORT` | ZAP listen address. See [listeners](listeners.md#zap). |
| `--mdns` | `$ZAP_MDNS`, else off | Advertise ZAP over mDNS and connect to the peers found. Never while ZAP listens on loopback. |
| `--no-mdns` | off | Keep mDNS off even when `--mdns` or `ZAP_MDNS` asks for it. |
| `--hooksDir` | `hooks` beside the data directory | JavaScript hooks. |
| `--hooksWatch` | `true` | Restart when a file under the hooks directory changes. Not on Windows. |
| `--hooksPool` | `15` | Interpreters kept for running hook handlers. |
| `--migrationsDir` | `migrations` beside the data directory | JavaScript migrations. |
| `--automigrate` | `true` | Write a migration when a superuser changes a collection over `/v1`. |
| `--publicDir` | `public` beside the binary | Parsed, but serves nothing: Base registers its own `GET /{path...}` route (the admin UI, or a `404` when the admin is off) before the binary checks for one. |
| `--indexFallback` | `true` | Parsed, but has no effect, for the same reason. |
| `-v`, `--version` | | Print the version. |
## serve flags
| Flag | Default | Meaning |
|---|---|---|
| `--http` | `127.0.0.1:8090`, or `0.0.0.0:80` when domains are named | HTTP address. While HTTPS is on, it only redirects and answers ACME challenges. |
| `--https` | none, or `0.0.0.0:443` when domains are named | HTTPS address. |
| `--origins` | `*` | Allowed CORS origins. |
## update flags
| Flag | Default | Meaning |
|---|---|---|
| `--backup` | `true` | Back up the data directory after replacing the binary. |
## cli flags
| Flag | Default | Meaning |
|---|---|---|
| `--url` | `$BASE_URL`, else `http://127.0.0.1:8090` | Base to call. |
| `--token` | `$BASE_TOKEN`, else the file `~/.config/base/token` (`$XDG_CONFIG_HOME/base/token` when set) | Token to send. Get it from IAM. |
| `--base` | none | Org to act in, sent as `X-Org-Id`. |
| `--format` | `table` on a terminal, else `json` | `table`, `json` or `yaml`. |
| `--mainnet` (`-m`), `--testnet` (`-t`), `--devnet` (`-d`), `--dev` | `$APP_ENV`, then `$BASE_ENV`, else local | Environment for `cluster`, `operator` and `status`. |
## Environment
A boolean accepts what Go's `strconv.ParseBool` accepts: `1`, `t`, `true`,
`TRUE` and their false counterparts. The exceptions are `S3_ENABLED` (`true` in
any case), `CLOUD_SQL_ENABLED` (exactly `true`) and `WAITLIST_OPEN` (`1`, `true`
or `yes` in any case).
### Core
| Variable | Default | Meaning |
|---|---|---|
| `DATA_DIR` | | Data directory when `--dir` is not given. |
| `BASE_API_PREFIX` | `/v1` | Where the API is mounted. `/v1/iam` and `/healthz` stay where they are. |
| `BASE_ENABLE_ADMIN_UI` | off | Serve the admin UI at `/`. It signs in through IAM. |
| `BASE_FRAME_ANCESTORS` | `'self'` | CSP `frame-ancestors` for the admin UI. |
| `IAM_DISPLAY_NAME` | empty | Display name of the `iam` provider in `auth-methods`. |
| `NATS_URL` | none | Publish record create, update and delete events to NATS. |
| `S3_ENABLED` | off | Store files in S3 rather than the data directory. |
| `S3_ENDPOINT`, `S3_BUCKET_NAME` (or `S3_BUCKET`), `S3_REGION`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_FORCE_PATH_STYLE` | | The S3 connection, read when `S3_ENABLED` is set. |
| `THUMBS_MAX_WORKERS` | CPUs + 2 | Thumbnails generated at once. |
| `THUMBS_MAX_WAIT` | `60` | Seconds a thumbnail request waits for a worker. |
| `FILES_DELETE_MAX_WORKERS` | `2000` | File deletions run at once. |
| `BASE_PRIVATE_MAX_TAG_BYTES` | `1048576` | Largest value in `/v1/private`, the per-user encrypted blob store. |
| `BASE_PRIVATE_MAX_USER_BYTES` | `104857600` | Total `/v1/private` bytes per user. |
| `BASE_GOJAVM_POOL_SIZE` | `8` | Interpreters for `/v1/functions`, which is how many functions run at once. |
| `TASKS_EMBED` | off | Run the Tasks engine inside Base. See [listeners](listeners.md#embedded-tasks). |
| `TASKS_EMBED_ADDRESS` | `<data dir>/tasks.sock` | Where the embedded engine listens. |
| `TASKS_NAMESPACE` | `default` | Namespace of the embedded engine. |
| `TASKS_ZAP` | none | A Tasks server to use over ZAP. |
| `TASKS_URL` | none | A Tasks server to use over HTTP. With no Tasks server, scheduled jobs run on timers inside the process. |
### ZAP
| Variable | Default | Meaning |
|---|---|---|
| `ZAP_ADDR` | none | ZAP listen address. `--zap` wins over it. |
| `ZAP_PORT` | `9999` | Port used on the HTTP host when no address is given. |
| `ZAP_MDNS` | off | The same as `--mdns`. |
| `ZAP_DISABLED` | off | Do not start ZAP. |
### IAM, orgs and KMS
The binary registers `plugins/org` when `IAM_ENDPOINT` or `IAM_URL` is set.
| Variable | Default | Meaning |
|---|---|---|
| `IAM_ENDPOINT` (or `IAM_URL`) | none | IAM origin. Turns on IAM sign-in and a Base per org. |
| `IAM_ADDRESS` | none | Where Base reaches the IAM service to check tokens and keys, when that is not `IAM_ENDPOINT`. A bare `host:port` is ZAP. |
| `IAM_CLIENT_ID`, `IAM_CLIENT_SECRET` | none | The IAM application Base presents when it calls IAM itself, and to KMS over HTTPS. |
| `KMS_ENDPOINT` (or `KMS_URL`) | none | Secrets store: `host:port`, `zap://host:port`, `https://...`, or `zap+mdns://_kms._tcp` to discover it. Contacted on first use. |
| `LUX_MNEMONIC` (or `MNEMONIC`) | none | Seed of the identity Base signs KMS requests with. Required for KMS over ZAP. |
| `KMS_ENV` | `prod` | Environment of the secrets Base reads. |
| `KMS_SERVICE_PATH` | `hanzo/base` | Derivation path of that identity. |
| `KMS_NODE_ID` | `hanzo-base` | Node name the KMS client presents. |
| `IDV_ENDPOINT` | none | Upstream for `/v1/idv/*`. Unset, `/v1/idv/status` answers `{"enabled":false}` and the rest `503`. |
### Replication
| Variable | Default | Meaning |
|---|---|---|
| `BASE_NETWORK` | `standalone` | `quasar` replicates writes to peers. |
| `BASE_PEERS` | none | Peers as comma-separated `host:port`. Replication listens only when this names one. |
| `BASE_SHARD_KEY` | none | Required with `quasar`: a field of the verified identity such as `user_id` or `org_id`, or `header:<Name>`. |
| `BASE_REPLICATION` | `1` | Members that hold each shard. |
| `BASE_NODE_ID` | `$HOSTNAME` | This member's id. Required with `quasar`. |
| `BASE_NODE_ROLE` | `validator` | `validator` or `archive`. |
| `BASE_ARCHIVE` | `off` | `s3://bucket/prefix` to archive frames. |
| `BASE_LISTEN_P2P` | `:9999` | Peer listener, bound exactly as written. The port must be a number. |
| `BASE_LISTEN_HTTP` | `:8090` | Read, but opens nothing. |
| `BASE_PEER_HTTP_ENDPOINTS` | none | Comma-separated `peer=url` pairs: where to send a write that a peer owns. |
| `BASE_PEER_HTTP_PORT` | `8090` | HTTP port assumed for a peer missing from that list. |
| `AWS_REGION`, `AWS_ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` | | Credentials for an `s3://` archive. |
| `BASE_ENCRYPT`, `BASE_TLS_CA`, `BASE_TLS_SERVER_CERT`, `BASE_TLS_SERVER_KEY`, `BASE_TLS_ALLOWED_SANS` | | Refused. Setting any of them stops startup with an explanation. |
### Plugins in the binary
The waitlist plugin is always registered and serves `/v1/waitlist/*`.
| Variable | Default | Meaning |
|---|---|---|
| `TURNSTILE_SECRET_KEY` | none | Cloudflare Turnstile secret for checking joins. |
| `WAITLIST_ADMIN_SECRET` | none | Secret for `boost` and `export`. |
| `WAITLIST_AWARD_SECRET` | none | Secret for `award`. |
| `WAITLIST_DEFAULT_SLUGS` | none | Comma-separated waitlists to create at startup. |
| `WAITLIST_ACCESS_CAPACITY` | `0` | Grant access to the top N entries by rank. |
| `WAITLIST_OPEN` | off | Give everyone access. |
| `POINTS_REFERRAL`, `POINTS_SHARE`, `POINTS_INVITE_SENT`, `POINTS_INVITE_CONVERTED`, `POINTS_SOCIAL`, `POINTS_HANZOD`, `POINTS_SIGNUP` | `10`, `2`, `1`, `5`, `15`, `25`, `0` | Points per event. |
The other three are off unless enabled.
| Variable | Default | Meaning |
|---|---|---|
| `BOOTNODE_ENABLED` | off | Serve the bootnode developer platform. |
| `BOOTNODE_API_KEY_SALT` | none | Salt for issued API keys. Required when `IAM_URL` is not a local host. |
| `BOOTNODE_ALLOWED_ORGS` | `hanzo`, `zoo`, `lux`, `pars` | IAM orgs allowed to sign in. |
| `BOOTNODE_K8S_NAMESPACE` | `bootnode` | Namespace for the resources it applies. |
| `IAM_URL`, `IAM_CLIENT_ID`, `IAM_CLIENT_SECRET` | | The IAM application for its OAuth2 code exchange. |
| `FRONTEND_URL` | `http://localhost:3001` | Redirect base when a request names none. |
| `COMMERCE_URL`, `COMMERCE_API_KEY` | none | Billing client. An empty key turns billing off. |
| `KUBE_APISERVER`, `KUBE_TOKEN`, `KUBE_INSECURE_SKIP_TLS_VERIFY` | | Kubernetes API, when not running in a cluster. |
| `KUBERNETES_SERVICE_HOST`, `KUBERNETES_SERVICE_PORT` | | Kubernetes API from inside a cluster. |
| `CALENDAR_ENABLED` | off | Serve the booking API at `/v1/calendar`. |
| `CALENDAR_SEED_HANDLE` | none | Create a bookable host with this handle at startup. |
| `CALENDAR_SEED_OWNER`, `CALENDAR_SEED_TZ`, `CALENDAR_SEED_TITLE`, `CALENDAR_SEED_LOCATION` | the handle, `America/Los_Angeles`, `Intro`, `https://meet.hanzo.ai/<handle>` | That host's details. |
| `CLOUD_SQL_ENABLED` | off | Serve `/v1/cloud-sql` and `/v1/meta`. |
| `CLOUD_SQL_META_URL`, `CLOUD_SQL_COMPUTE_HOST`, `CLOUD_SQL_PG_USER`, `CLOUD_SQL_PG_PASS` | none | The managed PostgreSQL service it drives. |
### Plugins you link yourself
The binary does not include these. A program that registers one reads its
variables.
| Plugin | Variables |
|---|---|
| `plugins/ha` | `BASE_LOCAL_TARGET`, `BASE_PEERS`, `BASE_STATIC_WRITER`, `BASE_NODE_ID` |
| `plugins/replicate` | `REPLICATE_S3_ENDPOINT`, `REPLICATE_S3_PATH` |
| `plugins/tasks` | `TASKS_ENABLED` (or `HANZO_TASKS_ENABLED`), `TASKS_ADDRESS` (or `HANZO_TASKS_ADDRESS`), `TASKS_NAMESPACE` (or `HANZO_TASKS_NAMESPACE`), `TASKS_QUEUE`, `TASKS_WORKER` |
| `plugins/pyvm`, `plugins/v8vm`, `plugins/wasmvm`, `plugins/starkvm` | `BASE_PYVM_POOL_SIZE`, `BASE_V8VM_POOL_SIZE`, `BASE_WASMVM_POOL_SIZE`, `BASE_STARKVM_POOL_SIZE` |
### Read by a dependency
| Variable | Meaning |
|---|---|
| `HANZO_SQLITE_RAMFS_DIR` | Directory the pure-Go encrypted SQLite codec decrypts into. It must be RAM-backed: `/dev/shm` is used when this is unset, and on macOS only a `tmpfs` mount qualifies. With neither, an encrypted database does not open. |
| `HOSTNAME` | Default replication node id. |
### The cli command
| Variable | Meaning |
|---|---|
| `BASE_URL`, `BASE_TOKEN` | Defaults for `--url` and `--token`. |
| `XDG_CONFIG_HOME` | Where the token file and `base/config.json` live. |
| `APP_ENV`, `BASE_ENV` | Default environment when no environment flag is given. |
docs/hooks.md
+163
View File
@@ -0,0 +1,163 @@
# Hooks
A hook is JavaScript that runs inside the Base process: code that runs when a
record is written, a route of your own, or a job on a schedule. The `base`
binary built from `examples/base` loads hooks. A Go program that embeds Base
loads them only if it registers `plugins/jsvm`.
## Where hooks live
Base loads every file directly inside the hooks directory whose name ends in
`.base.js` or `.base.ts`.
- The directory is `--hooksDir`. Without it, Base uses `hooks` beside the data
directory, so the default data directory `base_/data` gives `base_/hooks`.
- Subdirectories are not loaded. Keep shared code there and `require` it.
- A `.base.ts` file runs as plain JavaScript. A type annotation is a syntax
error.
- At startup Base writes type declarations to `types.d.ts` in the data
directory. An empty hook file gets a `/// <reference>` line pointing at it, so
an editor can complete `$app`, `routerAdd` and the `on*` hooks.
## An example
A migration creates the collection, a record hook trims text before it is
saved, and a route counts the records.
`base_/migrations/1_notes.js`:
```js
migrate((app) => {
app.save(new Collection({
type: "base",
name: "notes",
listRule: "",
viewRule: "",
createRule: "",
fields: [{ name: "text", type: "text", required: true }],
}))
})
```
`base_/hooks/notes.base.js`:
```js
onRecordCreateRequest((e) => {
e.record.set("text", e.record.getString("text").trim())
e.next()
}, "notes")
routerAdd("GET", "/v1/notes/count", (e) => {
const { label } = require(`${__hooks}/lib/label.js`)
return e.json(200, { [label()]: $app.countRecords("notes") })
})
```
`base_/hooks/lib/label.js`:
```js
module.exports = { label: () => "notes" }
```
Start Base next to those files and call it:
```sh
./base serve --http 127.0.0.1:8090
```
```sh
curl -s -X POST http://127.0.0.1:8090/v1/collections/notes/records \
-H 'Content-Type: application/json' -d '{"text":" buy milk "}'
{"collectionId":"hbc_3395098727","collectionName":"notes","id":"ot6ovxamu9s9ovi","text":"buy milk"}
curl -s http://127.0.0.1:8090/v1/notes/count
{"notes":1}
```
## How a hook file runs
Each file runs once, at startup, and registers handlers: `routerAdd`,
`routerUse`, `cronAdd`, and the `on*` functions such as `onRecordCreateRequest`
and `onBootstrap`. There is no `onServe`; add routes with `routerAdd`. The
handlers run later, on a pool of interpreters whose size is `--hooksPool`
(default 15).
**A handler cannot see the rest of its file.** Base compiles each handler from
its own source text, so a `const` at the top of the file does not exist inside
the handler. This route fails:
```js
const greeting = "hello"
routerAdd("GET", "/v1/gotcha", (e) => {
return e.json(200, { greeting: greeting })
})
```
```sh
curl -s http://127.0.0.1:8090/v1/gotcha
{"data":{},"message":"Something went wrong while processing your request.","status":400}
```
Put shared code in a module and `require` it inside the handler, as the example
does. Modules are CommonJS: `module.exports` out, `require(path)` in. A relative
path resolves from the process's working directory, not from the hook file, so
build the path from `__hooks`, which holds the absolute path of the hooks
directory.
Record hooks take collection names after the handler, as
`onRecordCreateRequest(handler, "notes")` does. With no names, the handler runs
for every collection. A handler calls `e.next()` to let the rest of the chain
run, including Base's own work; a handler that returns without calling it stops
the chain there.
Inside a handler, `$app` is the Base the event belongs to. With IAM on, that is
the org's own Base, so a hook reads and writes that org's data. Record and model
hooks run on every org's Base. `routerAdd` and `cronAdd` register once, for the
whole process.
The interpreter is goja, a JavaScript engine written in Go. Beyond the language
it has `require`, `console`, `process` and `Buffer`, and no other Node.js
modules. Make HTTP requests with `$http.send`. Everything else Base offers is
declared in `types.d.ts`.
## Changing hooks while Base runs
With `--hooksWatch`, which is on by default, saving any file under the hooks
directory restarts Base in place: the same process, running the new code.
Hidden directories and `node_modules` are not watched. When the hooks directory
is a symlink, Base watches the directory it points to, but it does not follow
symlinks inside it. Windows cannot restart this way; restart by hand there.
A file that fails to load is logged and skipped while watching is on. With
`--hooksWatch=false`, the same failure stops the process.
## Migrations
Put schema in migrations. They are JavaScript files in `--migrationsDir`
(default `migrations` beside the data directory, so `base_/migrations`), and
`serve` applies the pending ones in file name order when it starts. Each file
calls `migrate(up, down)`; `down` is optional.
| Command | What it does |
|---|---|
| `base migrate up` | Applies pending migrations. |
| `base migrate down [n]` | Reverts the last `n`. |
| `base migrate create <name>` | Writes an empty migration file. |
| `base migrate collections` | Writes a migration holding every current collection. |
| `base migrate history-sync` | Forgets applied migrations whose files are gone. |
With `--automigrate`, on by default, a collection change a superuser makes over
`/v1` is also written out as a migration file.
## Before you ship hooks
A hook runs with the full authority of the process. `$os` can run commands and
read and write files, and `routerAdd` can claim any path. Load only hooks you
wrote or read. [threat-model.md](threat-model.md) covers what that means for a
deployment.
A route added with `routerAdd` applies no collection rules. Check them yourself
with `$app.canAccessRecord(record, e.requestInfo(), collection.listRule)`. The
[hybrid search example](../examples/hooks-hybrid-search) does this for every
result it returns.
docs/listeners.md
+123
View File
@@ -0,0 +1,123 @@
# Network listeners
Every socket a Base process can listen on: what opens it, where it binds, and
how to change that. A `serve` with no flags and no environment opens two, both
on loopback:
```sh
./base serve
lsof -nP -a -p <pid> -iTCP -sTCP:LISTEN
```
```
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
base 62716 z 18u IPv4 0x570514825a9f5e0d 0t0 TCP 127.0.0.1:8090 (LISTEN)
base 62716 z 19u IPv4 0x26859a4d8573f408 0t0 TCP 127.0.0.1:9999 (LISTEN)
```
| Listener | Opens when | Binds by default | Change with |
|---|---|---|---|
| HTTP | `serve` | `127.0.0.1:8090`; `0.0.0.0:80` when domains are named | `--http` |
| HTTPS | `--https`, or domains named | `0.0.0.0:443` when domains are named | `--https` |
| ZAP | `serve`, unless `ZAP_DISABLED=true` | the HTTP host, port `9999` | `--zap`, `ZAP_ADDR`, `ZAP_PORT` |
| mDNS | `--mdns` or `ZAP_MDNS=true`, and ZAP is not on loopback | UDP `5353` on every multicast interface | `--no-mdns` keeps it off |
| Replication | `BASE_NETWORK=quasar` and at least one entry in `BASE_PEERS` | `:9999`, every interface | `BASE_LISTEN_P2P` |
| Embedded Tasks | `TASKS_EMBED=true` | unix socket `<data dir>/tasks.sock` | `TASKS_EMBED_ADDRESS` |
| KMS discovery | `KMS_ENDPOINT=zap+mdns://...`, at the first secret read | a port the OS picks, on every interface, plus mDNS | a `host:port` endpoint, which dials instead |
## HTTP and HTTPS
`serve` listens on `--http`, which is `127.0.0.1:8090` unless you say otherwise.
Name domains (`base serve example.com`) and the defaults become `0.0.0.0:80`
and `0.0.0.0:443`, with certificates from Let's Encrypt cached in
`<data dir>/.autocert_cache`. `--https <address>` turns HTTPS on without naming
domains; the certificate is then for the host in that address. While HTTPS is
on, the HTTP address only redirects to HTTPS and answers ACME challenges.
The API is under `/v1` (`BASE_API_PREFIX` moves it) and `GET /healthz` answers
at the root. With replication on, `GET /-/base/members` also answers, with no
credential required. The admin UI is served at `/` only when
`BASE_ENABLE_ADMIN_UI=1`.
## ZAP
ZAP is a binary protocol. The ZAP listener serves the same handler as HTTP: the
same routes, middleware, IAM checks and rules. It adds a port, not a way around
any of them.
- The address is `--zap`, else `ZAP_ADDR`, else the HTTP listener's host on
`ZAP_PORT` (default `9999`). `--http 127.0.0.1:8090` gives `127.0.0.1:9999`;
`--http :8090` gives `:9999`, which is every interface. With HTTPS on, the
host comes from the HTTPS address.
- An address that starts with `/`, `./`, `../` or `@` is a unix socket.
- If the address cannot be read or cannot be bound, Base logs an error and
keeps serving HTTP.
- The container image runs `serve --http=0.0.0.0:8090`, so inside a container
ZAP listens on `0.0.0.0:9999`. Publish only the ports you mean to.
- `ZAP_DISABLED=true` turns it off.
## mDNS
mDNS is off unless `--mdns` or `ZAP_MDNS=true` asks for it, and it stays off
when ZAP listens on loopback or a unix socket. `--no-mdns` keeps it off
whatever else is set. When it runs, Base joins the multicast groups
`224.0.0.251` and `ff02::fb` on UDP port 5353 on every multicast-capable
interface, advertises `_hanzo-base._tcp` under the host name, and connects to
the peers it finds.
## Replication
`BASE_NETWORK=quasar` replicates SQLite writes to the peers in `BASE_PEERS`.
With no peers it opens nothing.
- The peer listener binds `BASE_LISTEN_P2P` exactly as written. The default,
`:9999`, is every interface; `127.0.0.1:9999` is loopback. The port must be a
number: `BASE_LISTEN_P2P=127.0.0.1` stops startup with
`missing port in address`.
- Startup also stops without `BASE_SHARD_KEY`, or without `BASE_NODE_ID` when
`$HOSTNAME` is unset.
- Peers present no certificate. Whatever can reach this port can submit frames
for the shards this node owns, so `BASE_PEERS` and your network policy are the
whole trust boundary. [NETWORK.md](NETWORK.md) has the design.
- ZAP defaults to port 9999 as well. Replication starts first, so when both land
on one address, ZAP is the one that logs `address already in use` and stays
off. Give one of them another port.
- `BASE_LISTEN_HTTP` is read but opens nothing.
## Embedded Tasks
`TASKS_EMBED=true` runs the Tasks engine inside Base and listens for it at
`TASKS_EMBED_ADDRESS`, by default `<data dir>/tasks.sock`. A path is a unix
socket; anything else is `host:port`. When the engine fails to start, Base logs
nothing and runs scheduled jobs on timers inside the process. Rule out these
causes:
- A relative data directory, such as the default `base_/data`. The socket path
then starts with neither `/` nor `./`, is read as a TCP address, and cannot
bind. Give an absolute `--dir` or `TASKS_EMBED_ADDRESS`.
- No RAM-backed directory for the engine's encrypted store: `/dev/shm` on
Linux, or on macOS a `tmpfs` mount named by `HANZO_SQLITE_RAMFS_DIR`.
## KMS
The KMS client does nothing until Base first reads a secret. A `host:port` or
`zap://host:port` endpoint then dials that address and listens on nothing, and
an `https://` endpoint uses HTTPS. `zap+mdns://_kms._tcp` is the exception: to
find KMS, it starts a ZAP node that listens on a port the OS picks, on every
interface, and runs mDNS.
## Connections Base makes but never accepts
- IAM: HTTPS to `IAM_ENDPOINT` for signing keys and for `/v1/iam/*`, and
`IAM_ADDRESS` when set.
- Tasks: `TASKS_ZAP` over ZAP or `TASKS_URL` over HTTP. The client dials and
does not listen.
- NATS: `NATS_URL`, to publish record events.
- S3 for files (`S3_*`) and for the replication archive (`BASE_ARCHIVE`), SMTP
when mail is configured, and whatever a hook calls with `$http.send`.
- Replication: the peers in `BASE_PEERS`, and the HTTP address of the peer that
owns a write.
## Checking a running Base
`lsof -nP -a -p <pid> -i` lists every socket a process holds, listening or not.
docs/rules.md
+134
View File
@@ -0,0 +1,134 @@
# Access rules and identity
A collection has five rules: `listRule`, `viewRule`, `createRule`, `updateRule`
and `deleteRule`. They are all of Base's authorization. Base checks them in Go
on every `/v1` request. A list rule becomes part of the query, so a row the
caller may not see is neither returned nor counted.
## Rule values
| Value | Who passes | A caller who fails gets |
|---|---|---|
| `null`, the default | superusers | `403 Only superusers can perform this action.` |
| `""` | everyone, signed in or not | |
| an expression | callers for whom it is true | list: a page with no items; view, update, delete: `404`; create: `400 Failed to create record.` |
An expression is a filter over the record and the request:
```
@request.auth.id != ""
owner = @request.auth.id
status = "published" || owner = @request.auth.id
@request.body.role:isset = false
```
| Part | Forms |
|---|---|
| Compare | `=` `!=` `>` `>=` `<` `<=` `~` (contains) `!~` |
| Compare any of several values | `?=` `?!=` `?>` `?>=` `?<` `?<=` `?~` `?!~` |
| Combine | `&&` `\|\|` and parentheses |
| The request | `@request.auth.*` `@request.body.*` `@request.query.*` `@request.headers.*` `@request.method` `@request.context` |
| Another collection | `@collection.<name>.<field>` |
| Modifiers | `:isset` `:changed` `:length` `:each` `:lower` |
`?filter=` on list requests uses the same syntax.
Rules also decide what `?expand=` returns and what the `/v1/rest/{table}` reads
return. A route you add in a hook applies no rule on its own. Call
`$app.canAccessRecord(record, e.requestInfo(), collection.listRule)` for each
record, as the [hybrid search example](../examples/hooks-hybrid-search/hooks/search.js)
does.
## Where identity comes from
Base signs nobody in. It has no passwords, no sign-up and no sessions of its
own. A caller is whoever a Hanzo IAM access token says, sent as
`Authorization: Bearer <token>`.
Point Base at IAM to turn this on:
```sh
IAM_ENDPOINT=https://hanzo.id ./base serve --http 127.0.0.1:8090
```
With `IAM_ENDPOINT` set, Base:
- verifies every token against IAM's keys at
`$IAM_ENDPOINT/v1/iam/.well-known/jwks`, and treats a token that fails as no
token;
- ignores tokens signed any other way, including those an older Base issued;
- passes `/v1/iam/*` through to IAM;
- lists IAM as the one sign-in provider at
`GET /v1/collections/users/auth-methods`, with IAM's authorize URL and a PKCE
challenge;
- deletes `X-Org-Id`, `X-User-Id` and `X-User-Email` from every request as it
arrives. `X-Org-Id` is read first, as the org the caller asks for.
For the length of one request, the token becomes an auth record that is never
saved:
| Field | From the token |
|---|---|
| `id` | `sub` (else `preferred_username`): kept when it is 15 characters of `a-z`, `0-9` and `_`, padded with `_` when shorter, otherwise the first 24 hex characters of its SHA-256 |
| `email` | `email` |
| `name` | `name`, else `displayName` |
| `org_id` | the org the request acts in, when the auth collection has an `org_id` field |
Rules read these as `@request.auth.id`, `@request.auth.email` and so on.
**Superusers are the members of IAM's `admin` org**, and nobody else. Their
record is mirrored into `_superusers`; everyone else's into `users`. A superuser
passes every rule, manages collections and settings, and can act in any org. An
`admin` role inside an ordinary org is a different thing and grants none of
this.
**Each org has its own database.** Base takes the org from the token's
memberships and opens `<data dir>/orgs/<org>/data.db` for the request. Two orgs
never share a file, so a query in one cannot return another's rows. A token
that belongs to several orgs picks one with `X-Org-Id: <org>`. Naming an org
the token does not carry gets `403`, and so does a token with no org.
**Keys** stand in for tokens when no person is signing in. IAM issues them.
`sk-` keys are for servers. `pk-` keys are for web pages and reach only routes
marked public, such as record lists and views.
## Without IAM
Leave `IAM_ENDPOINT` unset and the `base` binary serves one Base with no orgs.
There is no sign-in route, so a caller without a token cannot get one: every
such caller is anonymous and none is a superuser. That is enough to build
against collections whose rules admit anonymous callers. Define the collections
in [migrations](hooks.md#migrations), because `/v1/collections` needs a
superuser.
Tokens Base signed itself still count, though, and `auth-refresh` renews them.
A superuser token left from an older version is one; [versions.md](versions.md)
shows how to revoke it.
Such a Base prints this when it starts:
```
(!) No superuser exists yet. Create the first one with:
./base superuser upsert EMAIL PASS
```
That command was removed. Superusers come from IAM.
## Close the open users collection
A new Base has a `users` collection whose `createRule` is `""`, so anyone can
add rows to it. Nobody can sign in as those rows, but it is still a public
write. Close it with a migration:
```js
migrate((app) => {
const users = app.findCollectionByNameOrId("users")
users.createRule = null
app.save(users)
})
```
After it runs, an anonymous `POST /v1/collections/users/records` gets `403`.
Rate limits and CORS have defaults to change too. See
[threat-model.md](threat-model.md).
docs/threat-model.md
+12 -9
View File
@@ -9,10 +9,12 @@ with the thing you do about it.
## The boundary
**Identity comes from Hanzo IAM and from nowhere else.** Base hosts no login,
stores no password and issues no token. A request is whoever IAM's signed token
says it is, verified against IAM's JWKS. `IAM_ENDPOINT` is required; a Base with
no IAM refuses to boot rather than falling back to something weaker.
**Identity comes from Hanzo IAM and from nowhere else.** Base hosts no login and
stores no password. A request is whoever IAM's signed token says it is, verified
against IAM's JWKS. `IAM_ENDPOINT` turns that on. Without it, the `base` binary
serves one Base in which every HTTP caller is anonymous, except one holding a
token Base signed itself, such as a superuser token left from an older version.
**Rotate those after an upgrade**; [versions.md](versions.md) shows how.
**A tenant is a file.** The org on a verified token selects
`{DataDir}/orgs/{org}/data.db`, and every read and write in `/v1` lands there.
@@ -34,11 +36,12 @@ a different authority and grants none of it.
## At rest
**The platform database is not encrypted by the driver.** `DefaultDBConnect`
opens `data.db` with pragmas only — no key. Per-org shards are different: they
open under a per-org DEK derived from the master key, SQLCipher under cgo and a
pure-Go codec envelope otherwise. **Put the data directory on an encrypted
volume.** Anyone who can read the file can read the platform Base with any
SQLite client.
opens `data.db` with pragmas only — no key. Per-org shards open under a per-org
DEK derived from a master key, SQLCipher under cgo and a pure-Go codec envelope
otherwise, but only when the org plugin has read that key from KMS, which it
does only when it is given an `IAMOrg`. The `base` binary gives it none, so its
org shards open unencrypted as well. **Put the data directory on an encrypted
volume.** Anyone who can read a file can read that Base with any SQLite client.
**Settings are plaintext JSON unless you set the encryption env.** SMTP
passwords, S3 secrets and OAuth2 client secrets live in `_params`. With
docs/versions.md
+96
View File
@@ -0,0 +1,96 @@
# Versions and upgrading
## What each tag serves
Tag numbers do not run in a straight line: two lines of work were tagged in the
same range. This is what each range serves, read from its code.
| Tags | Default API prefix | Sign-in |
|---|---|---|
| `v0.36.7-hanzo.1` to `v0.39.9` | `/api` | passwords, `base superuser` |
| `v0.39.10`, `v0.39.11` | `/v1/base` | passwords, `base superuser` |
| `v0.39.12`, `v0.39.13` | `/api` | passwords, `base superuser` |
| `v0.39.14` to `v0.39.17` | `/v1` | Hanzo IAM |
| `v0.40.0` to `v0.52.3` | `/api` | passwords, `base superuser` |
| `v1.0.0` | `/api` | Hanzo IAM |
| `v1.1.0` and later | `/v1` | Hanzo IAM |
From `v0.39.10` on, `BASE_API_PREFIX` overrides the prefix. On a password tag,
superusers and users sign in at
`<prefix>/collections/<collection>/auth-with-password`. IAM tags have no such
route.
The default data directory's name differs between tags too (`hz_data`, `data`,
`base_/data`), and so do the hooks and migrations directories beside it. Current
versions use `base_/data`. Pass `--dir`, `--hooksDir` and `--migrationsDir` when
you upgrade, and the names stop mattering.
## Where to get a version
| Channel | What is there |
|---|---|
| Go module | Tagged versions of `github.com/hanzoai/base`; `go list -m -versions github.com/hanzoai/base` lists them. |
| Source | `git clone --depth 1 --branch <tag> https://github.com/hanzoai/base`, then `CGO_ENABLED=0 go build` in `examples/base`. |
| Container images | [`ghcr.io/hanzoai/base`](https://github.com/hanzoai/base/pkgs/container/base), tagged with the version, with or without the `v`. Not every git tag has an image, so check before you pin. `latest`, `main` and `dev` are old builds. |
| GitHub releases | Binaries only for `v0.36.7-hanzo.1`, which serves `/api`. Later releases have none, so `base update` stops with `missing asset`. |
`go install github.com/hanzoai/base/examples/base@<version>` does not work: the
module's `go.mod` has a `replace` directive, and `go install` refuses those.
## Moving from /api and passwords to /v1 and IAM
A current Base opens an older data directory in place and migrates it when it
starts. Data written by `v0.36.7-hanzo.1` comes through with its collections,
rules and records.
**Paths.** Routes move from `/api/...` to `/v1/...`, and the old paths answer
`404`. The JavaScript client adds the prefix itself, so give it the origin:
```ts
import { BaseClient } from '@hanzo/base'
const base = new BaseClient('http://127.0.0.1:8090')
const page = await base.collection('notes').getList(1, 20)
```
Give it `http://127.0.0.1:8090/v1` instead and every call goes to
`/v1/v1/...` and fails with `404`. If Base is mounted under a longer prefix, set
the same value in both places: `BASE_API_PREFIX` on the server and `prefix` on
the client.
**Sign-in.** Password, OTP and OAuth2 sign-in are gone, and so is
`base superuser`. On first start the migrations drop the `password` column of
`_superusers` and the `_mfas`, `_otps`, `_externalAuths` and `_authOrigins`
tables. People sign in at Hanzo IAM and send its access token, and superusers
are the members of IAM's `admin` org. Set `IAM_ENDPOINT`; [rules.md](rules.md)
covers the rest.
**Tokens from before the upgrade.** Until `IAM_ENDPOINT` is set, Base still
accepts tokens it signed itself. A superuser token from the old version keeps
superuser access, and `POST /v1/collections/_superusers/auth-refresh` trades it
for a fresh one, so it never has to expire. Setting `IAM_ENDPOINT` ends that:
the same token then gets `401`. Without IAM, rotate the signing secrets, using
that token one last time:
```sh
curl -X POST http://127.0.0.1:8090/v1/collections/_superusers/rotate \
-H "Authorization: $OLD_TOKEN"
```
Every `_superusers` token signed before the rotation, `$OLD_TOKEN` included,
then gets `401`. Rotate `users` the same way if it held accounts.
**Ids in rules.** Rules carry over as written, but `@request.auth.id` now comes
from the IAM token (see [rules.md](rules.md#where-identity-comes-from)). A rule
such as `owner = @request.auth.id` matches only records whose `owner` holds the
id derived from that token. Old `users` rows stay, and nobody signs in as them.
**Hooks and migrations.** They load the same way. A hook file that calls a
removed sign-in hook, such as `onRecordAuthWithPasswordRequest`, fails to load,
and the log says so on the first start.
## Upgrading safely
Copy the data directory, run the new binary against the copy with `--dir`, and
check your collections, rules and hooks before you switch. `base update` cannot
do this for you while releases carry no binaries.
examples/hooks-hybrid-search/README.md
+65
View File
@@ -0,0 +1,65 @@
# Hybrid search with hooks
Search a collection by words and by meaning in one request. SQLite FTS5 ranks
the words with BM25, cosine similarity ranks stored embeddings against a query
vector, and reciprocal rank fusion merges the two rankings. Every result then
passes the collection's list rule.
| File | What it does |
|---|---|
| `migrations/1_docs.js` | creates the `docs` collection, an FTS5 table over its title and body, and the triggers that keep that table current |
| `migrations/2_sample_docs.js` | adds four documents with three-number embeddings |
| `hooks/search.base.js` | registers `POST /v1/docs/search` |
| `hooks/search.js` | ranks, fuses, and checks each result against the list rule |
## Run it
Build the binary as the [README](../../README.md#run-it) shows. Then, from
`examples/base`:
```sh
./base serve --http 127.0.0.1:8090 --dir "$(mktemp -d)" \
--hooksDir ../hooks-hybrid-search/hooks \
--migrationsDir ../hooks-hybrid-search/migrations
```
In another terminal:
```sh
curl -s -X POST http://127.0.0.1:8090/v1/docs/search \
-H 'Content-Type: application/json' \
-d '{"q": "rank", "vector": [0.1, 0.9, 0]}'
```
```json
{"items":[{"id":"5iy49tdy89q0knu","score":0.03252247488101534,"title":"Reciprocal rank fusion"},{"id":"ahov6ixwj26jsd8","score":0.01639344262295082,"title":"Vector similarity"},{"id":"mehis4xhns53hh6","score":0.015873015873015872,"title":"SQLite full-text search"},{"id":"zl12ppvcbbtofhv","score":0.015625,"title":"Access rules"}]}
```
"Reciprocal rank fusion" comes first because it is in both rankings: it is the
only document containing the word `rank`, and its embedding is the second
closest to the vector. Send only `q` to search by words, only `vector` to search
by meaning, and `limit` to cap the results (default 10, at most 50).
## How it works
- The triggers write to `docs_fts` in the same transaction as the write to
`docs`, whether the write came through the API, a hook, a migration or plain
SQL.
- The query is split into words and each word is quoted, so FTS5 reads
punctuation as text rather than as query syntax. Any word may match.
- Each ranking gives its top 100 documents `1 / (60 + rank)`, and a document's
scores add up.
- A route added in a hook applies no rules, so `search.js` calls
`canAccessRecord` with the collection's `listRule` for each result. Set that
rule to `@request.auth.id != ""` and an anonymous search returns no items.
## Limits
- Cosine similarity reads every stored embedding on every query. That suits
thousands of documents, not millions.
- `docs_fts` is matched to `docs` by `id`, so updating or deleting a document
scans `docs_fts`. The same scale applies.
- This example computes no embeddings. Store each document's vector from your
model in `embedding`, and send the query's vector, of the same length, as
`vector`.
- If you change the fields of `docs`, change the triggers in a new migration.
examples/hooks-hybrid-search/hooks/search.base.js
+8
View File
@@ -0,0 +1,8 @@
// POST /v1/docs/search {"q": "words", "vector": [0.1, 0.9, 0], "limit": 10}
//
// The handler is compiled on its own, so nothing declared at the top of this
// file is in scope inside it. The code lives in search.js and is required
// where it runs.
routerAdd("POST", "/v1/docs/search", (e) => {
return require(`${__hooks}/search.js`).search(e)
})
examples/hooks-hybrid-search/hooks/search.js
+109
View File
@@ -0,0 +1,109 @@
// Hybrid search over the docs collection.
//
// Two ranked lists: FTS5 matches ordered by BM25, and stored embeddings ordered
// by cosine similarity to the query vector. Reciprocal rank fusion merges them:
// a document scores the sum of 1 / (K + rank) over the lists it appears in.
// Every result then passes the collection's list rule, so the route returns
// what the caller could list from /v1/collections/docs/records.
const K = 60 // the RRF constant from Cormack, Clarke and Buettcher (2009)
const DEPTH = 100 // candidates taken from each list
// match quotes every word, so punctuation in the query is text to find rather
// than FTS5 syntax, and joins the words with OR, so any one can match.
function match(text) {
return String(text || "")
.split(/\s+/)
.filter((word) => word)
.map((word) => '"' + word.replace(/"/g, '""') + '"')
.join(" OR ")
}
function lexical(app, text) {
const query = match(text)
if (!query) {
return []
}
const rows = arrayOf(new DynamicModel({ id: "" }))
app.db()
.newQuery("SELECT id FROM docs_fts WHERE docs_fts MATCH {:query} ORDER BY bm25(docs_fts) LIMIT {:depth}")
.bind({ query: query, depth: DEPTH })
.all(rows)
return rows.map((row) => row.id)
}
function cosine(a, b) {
let dot = 0, aa = 0, bb = 0
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i]
aa += a[i] * a[i]
bb += b[i] * b[i]
}
return aa && bb ? dot / Math.sqrt(aa * bb) : 0
}
// semantic scores every stored embedding against the query vector. That is a
// full scan, which is fine for thousands of documents and not for millions.
function semantic(app, vector) {
if (vector.length === 0) {
return []
}
const rows = arrayOf(new DynamicModel({ id: "", embedding: "" }))
app.db().newQuery("SELECT id, embedding FROM docs").all(rows)
const scored = []
for (const row of rows) {
let embedding
try {
embedding = JSON.parse(row.embedding)
} catch (_) {
continue
}
if (Array.isArray(embedding) && embedding.length === vector.length) {
scored.push({ id: row.id, score: cosine(vector, embedding) })
}
}
return scored
.sort((a, b) => b.score - a.score)
.slice(0, DEPTH)
.map((s) => s.id)
}
function search(e) {
const info = e.requestInfo()
const body = info.body
const docs = e.app.findCollectionByNameOrId("docs")
const vector = []
for (let i = 0; body.vector && i < body.vector.length; i++) {
vector.push(Number(body.vector[i]))
}
const scores = {}
for (const list of [lexical(e.app, body.q), semantic(e.app, vector)]) {
list.forEach((id, rank) => {
scores[id] = (scores[id] || 0) + 1 / (K + rank + 1)
})
}
const limit = Math.min(Math.max(parseInt(body.limit, 10) || 10, 1), 50)
const items = []
for (const id of Object.keys(scores).sort((a, b) => scores[b] - scores[a])) {
let record
try {
record = e.app.findRecordById(docs, id)
} catch (_) {
continue // deleted since it was ranked
}
if (!e.app.canAccessRecord(record, info, docs.listRule)) {
continue
}
items.push({ id: id, title: record.getString("title"), score: scores[id] })
if (items.length === limit) {
break
}
}
return e.json(200, { items: items })
}
module.exports = { search }
examples/hooks-hybrid-search/migrations/1_docs.js
+31
View File
@@ -0,0 +1,31 @@
// The docs collection, and an FTS5 index over its title and body.
//
// Three triggers keep the index in step with docs. They run inside the write's
// own transaction, whichever way the write arrived: the API, a hook, a
// migration or plain SQL.
migrate((app) => {
app.save(new Collection({
type: "base",
name: "docs",
listRule: "",
viewRule: "",
fields: [
{ name: "title", type: "text", required: true },
{ name: "body", type: "text" },
{ name: "embedding", type: "json" },
],
}))
const statements = [
"CREATE VIRTUAL TABLE docs_fts USING fts5(id UNINDEXED, title, body)",
"CREATE TRIGGER docs_fts_insert AFTER INSERT ON docs BEGIN INSERT INTO docs_fts (id, title, body) VALUES (new.id, new.title, new.body); END",
"CREATE TRIGGER docs_fts_update AFTER UPDATE ON docs BEGIN UPDATE docs_fts SET title = new.title, body = new.body WHERE id = old.id; END",
"CREATE TRIGGER docs_fts_delete AFTER DELETE ON docs BEGIN DELETE FROM docs_fts WHERE id = old.id; END",
]
for (const sql of statements) {
app.db().newQuery(sql).execute()
}
}, (app) => {
app.db().newQuery("DROP TABLE IF EXISTS docs_fts").execute()
app.delete(app.findCollectionByNameOrId("docs"))
})
examples/hooks-hybrid-search/migrations/2_sample_docs.js
+15
View File
@@ -0,0 +1,15 @@
// Four sample documents with three-number embeddings, so the example runs
// without an embedding model. Real embeddings come from your model and are
// written to the embedding field like any other value.
migrate((app) => {
const docs = app.findCollectionByNameOrId("docs")
const samples = [
["SQLite full-text search", "FTS5 ranks matching rows with BM25.", [0.9, 0.1, 0.0]],
["Vector similarity", "Cosine similarity compares embeddings by angle.", [0.1, 0.9, 0.0]],
["Reciprocal rank fusion", "RRF merges ranked lists without tuning weights.", [0.5, 0.5, 0.1]],
["Access rules", "A list rule decides which records a caller may see.", [0.0, 0.2, 0.9]],
]
for (const [title, body, embedding] of samples) {
app.save(new Record(docs, { title, body, embedding }))
}
})