A Base its host opened for one org refused every token acting in
another, so a reader signed in with Hanzo ID could not comment on a
site whose Base is org hanzo's. Opening that org to any account would
hand the CRM to anyone, since its rules admit any signed-in caller.
A base collection now admits visitors when it says so, with
"visitors": true. A visitor reaches only its records and files; every
other route, and every collection that has not opted in, still answers
403. Its record carries role "visitor", admin false and its own org,
so `author = @request.auth.id` holds a row to its writer. Any other
collection's rules read a visitor as a guest, so a relation, expand or
filter from a visitors' collection carries it no further. Its writes
are held to ten a minute per Base, keyed by subject, whatever the
deployment limits. A superuser is never a visitor.
D12: a person's own views are CRM views rows with visibility private,
which only their maker lists, reads, changes or deletes, an org admin
included; an org view (visibility org, or empty for rows made before)
is every member's to read and its maker's or an org admin's to change.
IAM preferences keep only which view a person opens first.
A Base that installed the CRM earlier holds the old rules, and grow
never touched a rule. A grow document may now name the rules its own
replace (`replaces`): a collection still holding one of those exactly
takes the new rule, and a rule an org wrote is kept, so the next
install tightens the views of every Base that has them.
tests.Router compiles an app's routes for a test that asks several
requests of one app; the graph tests use it.
Red's review. The aggregate route reads columns directly, so it gated
only hidden fields: grouping an auth collection by email, or counting
its unique emails, answered addresses the records keep private, which
the list masks and the filter resolver will not compare. core.EmailGated
is now the one rule both ask: an auth collection's email, which a caller
who is not a superuser reads only where emailVisibility is set. The
aggregate route refuses it as it refuses a hidden field.
TestAPrivateEmailIsNotAFigure is red's proof inverted: the group and the
count are refused, and fail without the change.
A record's history showed the changes of a field the collection had
since removed. An org admin who removes a field means its values to be
gone from view, so a caller who is not a superuser no longer reads
them; _history keeps the row for a superuser.
The composite uniqueness check is a read before the write, which
PostgreSQL can race; that is now said where it is done and in LLM.md,
with the fix (a unique expression index from the dialect).
tests/data lacked twelve migrations, and every test app ran them before
its test began: about four seconds an app, which put core and apis past
go test's ten minutes on the runner, and past thirty once the timeout
was raised. The fixture now holds them (go run ./tests/fixture writes
it), and TestTheFixtureIsMigrated fails, naming the migration, while
one is missing. On one host core went from 267 s to 26 s, apis from
432 s to 23 s, and the whole suite runs in 51 s.
The delete tests gave a background goroutine 100 ms to remove a
record's files, which a slow runner did not always meet. BaseApp
counts those deletions and WaitFileDeletes waits for them: the tests
wait on the count instead of the clock, and ResetBootstrapState waits
too, so a Base that stops no longer drops a deletion in flight.
TestSQLRun expected a CREATE TABLE to report one affected row. SQLite
answers a statement that writes no rows with the connection's last
count, and that row was the last migration's _migrations insert; the
handler now reads 0, the number it changed.
go test runs with its default timeout again, and crm_test.go is gofmt'd.
Filter params were pasted in one at a time, so a value spelling another
param's placeholder was rewritten inside its own quotes and the second
value's quotes closed the first one's string: a bound value could add
conditions to the filter. They are now substituted in one pass over the
filter as written.
An org admin administers, and a member reads the schema of, only a Base
opened for its org (StoreKeyTenant): a Base deployed for no org belongs to
whoever deployed it. The org plugin names the org of every Base it opens.
A saved user's record never carries org authority: @request.auth.admin and
@request.auth.role answer only for the record a verified token mints, and
the two fields are system and hidden.
The emptiness figures of an aggregate are columns of its one query, so a
grouped request counts the groups it shows and runs one query however
many figures it asks for.
A stream the host routed to its org's Base is served by that live Base,
not the instance its grant was minted on.
Adding a field the collection did not have ran it through the single to
multiple conversion, which read the fresh JSON column as text. SQLite let
that pass; PostgreSQL refused it (`COALESCE(jsonb, '')`), so no select,
relation or file field with several values could be added to an existing
collection there, and the CRM module could not install. A new field has
no old shape to convert, so it is skipped.
A composite value is read as text like a JSON field, so comparing a whole
value with null is IS NULL and a LIKE compares text on both engines.
The engine corpus installs the CRM and filters its values on both.
A client draws a record by its schema, a field's label and a select
value's colour, so every member of the org reads the org's own
collections. Changing them stays an admin's, and Base's own tables stay
hidden from a member.
A module that indexes a column the org already indexed under another
name no longer fails the install: the index is there.
A base collection that turns History on gets a _history row for every
create, update and delete: the collection and record ids, the action, the
writer (an actor field stamped from the save's context) and each changed
field's from and to. The row and the record commit or roll back together.
id, autodate, actor and hidden fields are left out, a value over 4 KiB is
recorded only as changed, and one writer's updates to one record within ten
minutes fold into one entry.
GET /v1/collections/{collection}/records/{id}/history reads it, newest
first, for whoever the record's view rule admits.
A where clause is a filter as JSON: and, or and not over leaves of a field
path, an operator and a value. CompileWhere checks the path against the
collection (relations, back-relations, composite parts), admits the
operators the field's type takes, binds every value, and writes Base's own
filter language, so it runs under the same resolver and rules as a filter
written by hand. A negation is pushed down to the leaves, each of which
knows its opposite, an empty value included. Relative dates are computed
in the leaf's time zone and week, @me is the caller, and inSegment
expands a saved segment the caller may read.
The records list takes ?where= beside ?filter=, after the list rule.
GET /collections/{c}/aggregate answers count, sum, avg, min, max,
countUnique, countTrue/False and the empty counts and percentages, once
or per value of one field, over the ids the list rule, filter and where
select, so a record joined twice is counted once.
A module is a named import document, and installing one is a grow:
core.GrowCollections creates what is missing and adds the fields, select
values and indexes an existing collection lacks, in one transaction,
changing nothing that exists. A relation names its collection by name.
Installing twice changes nothing.
modules/crm carries the thirteen collections: people, companies,
opportunities, candidates, applications, lists, list_members, segments,
activities (one row per source id), notes, tasks, attachments and the
org's shared views. POST /v1/modules/{name} installs, GET /v1/modules
lists, both for a superuser or the org's admin, who can never reach a
collection only a superuser manages this way.
The _view* system collections are retired. Nothing read them, and team
views are the ordinary views collection now.
Seven field types a CRM record needs, each one JSON column whose parts a
filter, a sort or an aggregate reaches by name: currency (amountMicros,
currencyCode), emails, phones, links, address, fullName and actor, plus
rating. Empty is NULL. Emails, phones and links normalize what is typed and
can keep their primary part unique. An actor is stamped by the server from
the writer the save's context states; nothing a request sends sets it.
Every field carries a label, description, icon and position, a select's
values carry a label, colour and position, and a base collection carries a
label, plural, icon, description and an opt-in history flag.
Editor HTML is sanitized before it is written.
A rule reads the caller's authority in the org it acts in:
@request.auth.admin and @request.auth.role, off the token. An IAM-mirrored
auth record is never saved, so its fields are now read off the record
rather than joined against a row that is not there, which also makes
@request.auth.org_id match for the first time. A host that opens one Base
per org names it, and a token acting in another org is refused there.
zip's registration is a verb method on the group now, so an operation reads as
the group answering it rather than a function taking the group, and a route that
is not a typed operation says which method it answers with. The package-level
form is gone, along with the Router interface it existed to accept any target
through.
Co-authored-by: Hanzo Dev <[email protected]>
NOTICE keeps the MIT copyright line and points at LICENSE.md. The
realtime handshake is CONNECT on both sides, and the docs now say so.
Co-authored-by: Hanzo Dev <[email protected]>
Auth is Hanzo IAM and nothing else, so the registry a provider name was
once checked against is gone, and with it every check. A collection
could declare a google login and save clean, while auth-methods answers
iam whatever that collection stores. One name validates now, and it
lives in core beside the check rather than as a literal in apis.
The tests describing the removed registry are rewritten around the
provider that exists, fixtures included. TestAnchorStatusAfterClose
demanded an unchanged anchor height across a Close that joins the
anchor goroutine, so a submit in flight still landed and it failed 4
runs in 60; it now asserts the height cannot rewind.
Co-authored-by: Hanzo Dev <[email protected]>
Two conflicts, both version pins. go.mod takes the branch's
hanzo.ai/tygoja v0.0.3 — plugins/jsvm imports the vanity path, and the
old github.com/hanzoai/tygoja line no longer resolved it — over main's
newer github.com/hanzos3/go v1.0.2. ui-react/package.json takes the
branch's exact pins at main's newer numbers, and the pnpm override moves
with the dependency it overrides rather than holding @hanzo/ui two
releases back.
The branch tip did not compile on its own: it removed tools/auth but left
core/collection_model_auth_options_test.go importing it to exercise the
InitProvider it also removed. That test goes with the code it tested.
@tanstack/react-router pins 1.170.33 because router-plugin 1.168.36 peers
on ^1.170.33; exact pins do not float up to meet a peer the way the
ranges they replace did.
go vet ./... clean, ui-react builds and typechecks, dist rebuilt.
Co-authored-by: Hanzo Dev <[email protected]>
Base shipped 32 OAuth2 provider implementations and a discovery branch
that enumerated them whenever IAM was not configured -- a fallback that
opens exactly when the one auth source is missing.
The login routes that consumed them were already gone, so /auth-methods
advertised identity providers with live authorize URLs and PKCE
challenges that nothing could complete.
InitProvider and checkProviderName go with them: a collection can no
longer instantiate a provider, so the stored config is inert rather than
a wire waiting to be reconnected.
NewBaseCollection takes the id as an optional argument and nothing forbids it
matching the name, so callers set it that way to keep a stable, readable id
across deployments. The name/id collision check then looked up any collection
whose id equals the new name and refused if it found one — including the
collection being saved.
So such a collection was created happily, because nothing existed yet to collide
with, and could never be changed afterwards. Every later schema edit — one field,
one index — failed on restart with "The name must not match an existing
collection id", describing a collision with itself. A deployment either stayed on
the old schema or stopped booting.
Found from the other end: luxfi/bank creates every collection as
NewBaseCollection(name, name) and its daemon would not start against an existing
database once a new index was added to accounts.
The check keeps its real job — another collection's id is still not available as
a name, or FindCollectionByNameOrId would have two answers — and that half is
asserted separately, since it passed before this change and has to keep passing.
Co-authored-by: Hanzo Dev <[email protected]>
NewBaseCollection takes the id as an optional argument and nothing forbids it
matching the name, so callers set it that way to keep a stable, readable id
across deployments. The name/id collision check then looked up any collection
whose id equals the new name and refused if it found one — including the
collection being saved.
So such a collection was created happily, because nothing existed yet to collide
with, and could never be changed afterwards. Every later schema edit — one field,
one index — failed on restart with "The name must not match an existing
collection id", describing a collision with itself. A deployment either stayed on
the old schema or stopped booting.
Found from the other end: luxfi/bank creates every collection as
NewBaseCollection(name, name) and its daemon would not start against an existing
database once a new index was added to accounts.
The check keeps its real job — another collection's id is still not available as
a name, or FindCollectionByNameOrId would have two answers — and that half is
asserted separately, since it passed before this change and has to keep passing.
Co-authored-by: Hanzo Dev <[email protected]>
Six env flags, two conventions, and both failed silently against the other.
Five read `== "true"`, so BOOTNODE_ENABLED=1 left bootnode off and
CALENDAR_ENABLED=1 left the calendar off. One read `!= "1"`, so
BASE_ENABLE_ADMIN_UI=true served no admin. And ZAP_DISABLED read `!= "true"`, so
an operator who wrote ZAP_DISABLED=1 to stop the listener watched it come up
anyway. In every case the flag was accepted, ignored, and never mentioned again.
This tree already answers "read a boolean from a string" four times, with
strconv.ParseBool (tools/filesystem, tools/search, tools/router,
core/settings_query). The hand-rolled comparisons were the deviation, so
osutils.Bool is that answer given a name and a default, and all six sites read
it. Strictly more permissive: every spelling that worked before still works, and
the ones that silently did nothing now do what they say.
An unparseable value yields the default rather than its opposite — someone who
typed something meaningless has not asked for the other thing.
Coverage: plugins/zap 5.9% -> 23.5%, and DefaultConfig is pinned including the
case that started this (ZAP_DISABLED=1 must disable) and a garbage ZAP_PORT,
which must keep 9999 rather than quietly binding somewhere else.
Co-authored-by: Hanzo Dev <[email protected]>
The admin had no storage surface at all: nothing in ui-react named an engine, a
file or a size, so the one question a growing Base raises — how much is in here
and where is it — could only be answered by finding the process's data
directory.
`/v1/database` answers it, and Settings -> Data renders it. core.DescribeDatabase
measures the engine, the files while the app holds them, and a count per
collection through CountRecords, so both engines answer the same questions and
the answer has one shape. A collection the engine refuses to count reads as
unknown rather than as empty, and is logged, so one broken view leaves the rest
of the report standing rather than taking the page down.
core.ReclaimDatabase is the one thing the page can DO, and it does something
real: it rewrites both databases without the pages deleted records left behind,
folds the log back into them, refreshes the statistics, and returns the size
either side so the page reports what was freed rather than claiming something
was. Driven in a browser it took a database from 9.0 MB to 1.1 MB with the
record counts unchanged.
The rewrite goes BEFORE the fold. A VACUUM lands through the log like any other
write, so folding first leaves the whole rewrite sitting in the log and the app
holding more than it started with — 2,068,480 bytes to 2,665,912 that way, and
to 593,920 this way. Both orders shrink something, which is why
TestDatabaseReclaim measures each database separately: an earlier version was
satisfied by 8 KB off the auxiliary log and went on passing with the rewrite
removed entirely.
There is no switch, because whether an instance belongs on a server is the
host's question and Base stopped having an opinion about it. The page states
which of the two it is on and what the other would mean, and it does not name a
backend Base cannot open — a disabled control for one would be the datastore
tier constant again, in a place a customer can see.
core.LocalDatabase is now the single place that asks whether the app holds the
database; backup asked it inline and now asks it there. tests/engine gains the
cross-engine cases, which CI already runs against PostgreSQL 18 — the LLM.md
line saying it does not was two releases stale. The third copy of a byte
formatter went into ui-react/src/lib/format.ts and the two existing pages now
read from it.
The wire carried another vendor's namespace in its error codes (PGRST1xx) and
in the media type a caller sends to claim a single row. Both ends are ours, so
it said someone else's name for no reason a caller could act on.
Two vocabularies meet in a fault and the split is the point: a constraint the
STORE refused answers with the store's own SQLSTATE, because every SQL engine
already agrees on how to say that; a request the WIRE would not accept answers
with a word of ours — bad_query, bad_body, not_one_row, refused, not_found,
store_failed — because that is a fact about this API and nobody else gets to
name it. Cardinality is asked for with application/vnd.hanzo.object+json.
LLM.md described the product as a translation of somebody else's seven
services, mapping their names onto ours, which is both wrong (their auth is not
Hanzo IAM) and backwards. It says what Base is: one binary, one file, the
surfaces it serves, and rules enforced here in Go rather than pushed down.
plugins/zap/forward.go still called itself additive to four native message
types that plugin.go had already removed — one node, one way in, and the table
wire arrives over it like every other route.
An extension's hooks were stated once, on the Base the process started with,
so a write to a tenant's Base fired nothing an extension had asked for. The
file says what to do when a record is created; it does not say where, and the
where comes with the event.
So a declaration is recorded as something that can be stated on any Base, and
stated on each as it is constructed — the same place realtime is bound, for
the same reason. A hook still sees only the Base it fired on.
This grants an extension nothing it did not have. A file on disk is the
operator's, and it already ran with the process's authority; what changes is
which data its hooks see. An extension written when there was one Base now
runs against each, which is a behaviour change and is the thing to know.
Routes and crons an extension declares stay on the Base that loaded it: a
route is registered once on a router, and a schedule belongs to the process.
A tenant's Base was bootstrapped and never served, so nothing it scheduled
ever ran — its logs were never cleaned and its database never optimised.
Co-authored-by: Hanzo Dev <[email protected]>
Both halves of this existed and neither was a function: one runtime knew how
to invoke by name and bound no host, the other bound every host and could not
invoke. They are joined through the runtime interface that was already here
and used by nothing outside its own tests.
A function lives in a collection, so it is the tenant's own file, its rules
are the rules, and the two doors — the collection and /v1/functions — refuse
in the same words because there is one rule. The source is the server's: a
caller who may see that a function exists does not get what it says.
What a function may ask for is two reads of the Base it is running in, and
that is the whole host. Not the app, not a process, not a file, not the
network — a name bound in a first version is bound for good, so the list is
what a function needs and nothing anticipated.
It reads as its caller. The collection comes from the Base the credential
already resolved, and the rule from the caller's own identity, so a function
renders what the caller could have read and is never a way around them. The
payload does not answer a rule: `@request.body` is about a write, and letting
an invocation body speak for it would let a caller satisfy a read rule by
naming a field they sent.
Invoking counts under its own action, because running code is the cost.
A function that does not finish is stopped.
Co-authored-by: Hanzo Dev <[email protected]>
A certificate surface sat beside the transport, complete enough to read as a
defence and reached by nothing. It could not have been reached: the transport
holds one configuration for every peer, so one name verifies against one host
and fails against the rest, and the pinning it did was written against chains
the dialling side never has — it would have refused every outbound connection
the first time it ran. Its own comment named a plugin that issues the
certificates, and there is no such plugin.
So the names it published refuse now, for the reason the encryption name
does: an operator who mounts a keypair, sets one, and sees a clean start has
been told peers authenticate. What is true is stated instead — every host in
the peer list may submit frames for any shard this node owns, so keep it to
peers of one service in one namespace and decide what reaches the peer port
with a NetworkPolicy.
The shard key names its source in its own value. It had fallen back to a
request header for any key, so the identity a request routed by was the
caller's to assert; that this did not reach the two documented keys was an
accident of another package deleting those two header names first. A key is
read from the verified identity unless it says otherwise.
Co-authored-by: Hanzo Dev <[email protected]>
A collection's token secret is half the key every token it issues is signed
with. It was stripped when a collection was rendered and read back when one
was submitted, so a caller could name the key by sending it — through update,
and through an imported document.
Reading is now refused where rendering already was: the model keeps the
secrets it holds and takes none from a document, so create, update and import
are one statement rather than three.
Replacing one is a verb that carries nothing, and it replaces all of them at
once. A token signed before is refused after.
Co-authored-by: Hanzo Dev <[email protected]>
A subscriber registers on the broker of the Base its request resolved to, and
for anyone in an org that is the org's Base. The broadcast hooks were bound
once while building the router, on the process's Base, so a write to a tenant
Base fired hooks that had no broadcaster and nothing was ever sent. The stream
stayed open and the subscription succeeded.
Hooks are stated where a Base is constructed now, beside the two registrars
that already fire per tenant, through a list an app binds as it is built.
Routes and hooks are different things and only one of them was a router's
business, so the router registers the two endpoints and nothing else — the
process's Base binds once because there is only one place that binds.
Delete named its Base separately from the one its hook fired on, and that name
was captured while building the router. Its rule therefore ran against a Base
where a tenant's collection does not exist, so any rule that has to query
answered no and the delete was dropped without a word. It is the Base the hook
fired on.
Co-authored-by: Hanzo Dev <[email protected]>
A builder that can fail while rendering SQL has only a string to say so with,
and a string means whatever the SQL around it makes it mean. So the fallible
work moved to construction: the multi-match forms assemble their aliases and
inner comparison and return an error, a subquery says what it is missing, and
both render methods are pure formatting with no branch left to take.
The empty set is now expressible in exactly one place, where it discards the
comparison it belongs to rather than sitting inside it.
The aliases were regenerated on each render, so the count query and the rows
query described the same filter with different SQL. They are assigned once.
An address under a Base names a user, and the check for whose it is applied
only where the path stopped there. It applies wherever the user appears, so a
longer address is the caller's own by the same rule as a short one.
Co-authored-by: Hanzo Dev <[email protected]>
The CLI printed a fixed name in its examples and help text while already
taking that name as a parameter, so `operator` and `rpc` now interpolate what
they were handed, the way `cluster` already did.
The config file's per-env URL block goes with it. Nothing read it: EnvURLs
computes a service address from the environment's domain suffix, which is
what every command actually calls, so the block was a second unasked answer
to where a service lives. An older config still loads — an unknown key is
ignored — and its env block is dropped on the next write. The default org is
no longer a name.
The Tasks queue is named for this repo rather than written out three times.
Fixture hostnames follow the shape the operator emits, which one of those
files was already using.
crdt/privacy_fhe.go cited a PDF that is not in the tree; the two figures it
backed are in the design doc it already points at.
Co-authored-by: Hanzo Dev <[email protected]>
A Base is embedded SQLite. That is what `base serve` runs, what a local
developer gets, and what every tenant, org and user shard is. There is now no
setting for it and nothing to misconfigure.
Base can still open a server, and the /v1 data plane is the same either way.
Config.DataDSN and AuxDSN name one; empty is embedded. They forward to fields
BaseAppConfig already had, so this moves where the question is asked rather
than adding a way to answer it.
Whether an instance belongs on a server, and which one, is a question about a
deployment. Whoever places the instance answers it — cloud embeds Base as a
library and pools a Base per tenant, so it answers per tenant. Running that
server is a host's work: provisioning, upgrades, backups, keeping it alive.
Base having an opinion about it is what made the binary look like it needed
one.
So BASE_DB_TIER, BASE_DB_URL and ResolveStorageTier are gone, and with them a
datastore tier constant that was declared here and never implemented. OLAP is
a different access pattern and belongs where it is built, not as a reserved
word in this package.
Nothing in the fleet, in universe or in cloud set any of them.
Co-authored-by: Hanzo Dev <[email protected]>
Three things, all the same shape: SQLite's guarantees were being assumed on
an engine that does not make them.
A backup archives the data directory, which is where the database lives on
SQLite and not on Postgres. There it returned 204 and produced an archive
with no database in it, and restore then failed inside a goroutine after the
204 had gone out — so an operator saw successful backups, a populated list,
plausible sizes, and had nothing recoverable. It refuses on an engine whose
database is not in that directory, before extracting anything, and says where
to take the backup instead. Refusing beats shelling out to pg_dump: that adds
a binary that has to out-version the server, and its output cannot be restored
by the path we have, so it would be a second restore mechanism.
What stopped a restore is now retrievable. It is recorded in core, so a CLI
or a hook caller sees it too, and GET on the restore address renders it: 202
while running, 200 and the reason when stopped, 404 when this process has not
tried. POST still answers 204, because the admin client reads a body on any
other 2xx.
A serialization failure or a deadlock is a transient and is retried at the
transaction boundary, which is the only place a retry means anything — the
second statement of a deadlocked transaction reports 25P02, so retrying a
statement spends the attempt and loses the evidence. The transaction watches
its own statements for the two SQLSTATEs rather than reading the returned
error, which by then has been wrapped down to a message string. Batch had to
become re-runnable for this: it accumulated results outside the transaction.
Retrying turns a transient into a success and does not pretend to fix a
convoy: with a few concurrent opposing batches the loss goes to zero, and
with thirty pairs over six rows every pair collides and no budget clears it.
SQLite keeps the same executions at the same intervals.
Co-authored-by: Hanzo Dev <[email protected]>
Three things kept the sql tier from being the tier it is described as.
Settings encryption refused to boot. The ciphertext is base64 and the column
is jsonb, so Postgres rejected the insert while parsing it as JSON. It is
marshalled as a JSON string now and unwrapped on read. This changes the
stored shape: a legacy row is read as-is and rewritten on the first settings
write, after which an older binary refuses to start rather than misreading
it. Forward-only, and loud in the one direction that is not.
Every schema change caused a burst of failures. A pooled connection holds a
prepared plan whose result type the change invalidates, so the next read on
that connection fails once. Reads across twenty add-a-field rounds went from
83 of 160 to 160 of 160. The exec mode is describe_exec, which pgx documents
as the one that stays safe when the schema changes underneath it; it costs a
parse per execution, worst measured on a local socket and pipelined into the
same round trip over a real network.
The write pool was capped at one connection on every engine. That is SQLite's
single-writer rule and Postgres does not need it. The cap is asked of the
connection's own driver now, so the aux database answers for itself.
Lifting it makes lock-order deadlocks reachable in one process, where a single
connection had been serialising them away. Retrying a serialization failure is
what turns concurrent writes from possible into correct.
Co-authored-by: Hanzo Dev <[email protected]>
Base generates its schema through the dialect, so a column lands as a real
boolean, numeric or jsonb. It then compared those with SQLite's spellings.
Three of them differ, and each is now the dialect's to state (orm v0.6.25):
Like — only SQLite's LIKE folds case. A search for "ITEM" matched one row
of five on Postgres and all five on SQLite, with no error either
way. Postgres gets ILIKE; not the nocase collation, which PG17
refuses for LIKE outright.
Bool — 1 and 0 are not booleans outside SQLite. Quoted on Postgres, so one
literal reads as a boolean beside a boolean column and as text
beside a value read out of a JSON document.
Number — a value read out of a JSON document is text wherever columns are
typed, so it is read back as a number where it is compared to one.
The cast is guarded: a value that is not a numeral fails to match
rather than raising, which is the answer SQLite already gives.
The cast applies only to values that came out of JSON, marked at the two
extraction sites, so a real numeric column and the geoDistance result are
left alone.
Filters return the same rows on both engines now, with one residual the
filter language itself does not settle: a JSON number compared to a text
literal. SQLite's answers are unchanged.
Co-authored-by: Hanzo Dev <[email protected]>
LLM.md is tracked here and this repository is public, which makes every note
in it a published note. It had been written as a lab notebook: what each part
of the code does, and alongside that, where it does the wrong thing, with the
line number.
The engineering stays. What the dialect covers, how the tiers select, which
driver opens SQLite and why exactly one may, where the mount prefix applies,
what the admin UI is built on — all unchanged, and that is most of the file.
What changes is how a defect is written. Where the code now holds a contract,
the file states the contract and not the road to it. Where it does not yet,
the file says nothing, and the detail lives in the project's private record
instead. Nine such notes moved; the same edit went through eight source
comments that had grown the same voice.
Three notes were simply stale and now match the code: the org shards are
encrypted per org through sqlite.OpenDB, the auth identifiers resolve, and
update checks its post-image.
Co-authored-by: Hanzo Dev <[email protected]>
The rule and the counters moved onto the Base the process serves from; the client
KEY did not, and that turned the previous fix into a cross-tenant denial of
service. checkRateLimit keys on RealIP, and RealIP asked e.App which proxy
headers to believe — the tenant's Base by then, whose TrustedProxy is empty and
can never be anything else, since only a superuser writes settings and a tenant's
Base has none. Behind an ingress every authenticated request fell back to the
socket peer, which is the ingress for all of them: one bucket for the estate, so
one tenant spending its budget locked out every other. Measured: alpha spends a
3-per-minute budget and beta, arriving from another address, is refused 3 of 3.
Rather than teach the limiter a second way to find that Base, core.RequestEvent
now names it — Deployment() — and everything that answers a question about the
process rather than about a tenant's data reads it there. RealIP, the rate limit
and its counters, the batch limits, and the activity log.
The log had the same bug and worse consequences. logRequest read e.App, so an
authenticated request wrote its audit row into the tenant's own file and the
operator's log held only traffic that never authenticated. It also wrote
RequestURI() whole, and a key arrives in ?key= on a request that carries no
Authorization header, so a live secret key sat in _logs in plaintext for
Logs.MaxDays, was served by GET /v1/logs, and went into every backup. The query
is redacted for the parameters this deployment reads a credential out of. And a
keyed request sets no e.Auth, so every action a service key took was attributed
to nobody; the subject is what both doors publish, so a subject with no record is
exactly a key.
Batch read Enabled off the tenant too, and a freshly opened Base answers false,
so an operator who turned batching on had it silently off for everyone with a
token.
RemoteIP reports no address where it cannot read one, instead of the zero Addr,
whose expanded spelling is the literal string "invalid IP". A ZAP-forwarded
request carries no socket peer, so the whole leg was counted as that one client.
The comments on this door carried another company's brand 72 times — their
product name, their client library, their console. This is our canonical
repo; the surface is ours and reads as ours.
What the prose says now is what the code DOES: a REST data wire with one
table per path and filters in the query string, answering a bare array with
the count in Content-Range. Every fact that made those comments worth having
survives — why a value binds rather than renders, why an unaligned offset is
refused, why neq is `<>` and not the null-safe form — because none of it
depended on whose wire it is.
The wire VALUES stay exactly as they are: PGRST116, PGRST100,
application/vnd.pgrst.object+json, SQLSTATE 23505. Those are not branding,
they are the contract — a client branches on them, and renaming one would
break every caller while changing nothing about who owns the surface.
`owner = @request.auth.id` is the first rule anyone writes, and for a caller who
was not signed in it returned the rows it exists to protect.
Three reasonable steps composed into an inversion. `@request.auth.id` resolves to
the identifier NULL when there is no auth. The filter compiler counts the literal
"null" as an EMPTY identifier. Equality against an empty identifier is rewritten
to `(owner = '' OR owner IS NULL)` so a missing value can still be compared. Each
is defensible alone; together they published every row whose owner was empty or
NULL to anybody who asked. Postgres answers the opposite — `owner = NULL` is
NULL, NULL is not true, the row is hidden — so a ported policy meant the reverse
of what it said.
The identifier could not carry this. Marking it non-collapsing only moves the
problem: `IS NOT DISTINCT FROM NULL` still matches a genuinely NULL column, and
it fixes `=` while leaving `!=` inverted. So the absent identity poisons the
comparison it appears in, at AfterBuild, after both operands are combined. That
settles every operator at once — `owner != @request.auth.id` excludes everything
too, exactly as `owner <> NULL` does — and it cannot be defeated by which side of
the comparison the rule author wrote it on, because AfterBuild is honored from
either operand.
The test is behavioural rather than an assertion about generated SQL, because the
property worth pinning is "an anonymous caller sees nothing", not "the compiler
emits this string". It seeds a collection with one unowned row and one owned row
and reads it with no credentials; before this change the response carried
UNOWNED_ROW and totalItems 1. The owned row is in there so that a fix which
simply returned nothing at all would not pass.
One behaviour changes on purpose: a rule written as `@request.auth.id = ""` to
mean "allow anonymous" now denies. That spelling never matched Postgres, where it
is `NULL = ''` and therefore false, and Base already has an honest way to say
public — the empty rule string. `@request.auth.id != ""` is unaffected and still
means "is signed in".
core, core/validators, apis and tools/search all pass.
Co-authored-by: Hanzo Dev <[email protected]>
Two halves of one gap.
The detection was SQLite's wording alone -- a literal match on "unique
constraint failed" -- so on Postgres a duplicate never normalized into a
field error and surfaced as a 500 instead of a 400. IsUniqueViolation reads
both engines now, including the bare SQLSTATE. Postgres names the INDEX
rather than the columns, so the field is recovered from the index name, which
Base builds out of the column; the match is on a whole underscore-delimited
token, because a substring would let a field named "e" claim every index
containing the letter. There is no Postgres in CI, so that half is proven
against its wording in a unit test.
On /rest/v1 a failure now renders PostgREST's {message, details, hint, code}
rather than Base's {data, message, status}, and a unique violation becomes
409 with code 23505. That is not cosmetic: handling a duplicate is ordinary
application logic, and every `error.code === '23505'` branch in a migrated
app is dead against a body with no code field. Tested against a real
violation of idx_unique_demo2_title, not a simulated one.
Base's own door is unchanged and a test says so -- the same duplicate still
answers validation_not_unique there. The PostgREST body belongs to the
PostgREST wire, not to the store.
BASE_DB_TIER=sql opened PostgreSQL and then spoke to it in SQLite: strftime,
sqlite_master, json_extract, json_each, an INTEGER column holding an int64, an
outer join with no ON, backticked identifiers, and a rowid that only one engine
has. The tier picked a backend the queries could not reach.
Every one of those now comes from hanzoai/orm/dialect, resolved once from the
driver the data DB was opened with and reachable as app.Dialect(). Base holds no
switch on the engine — it asks for the JSON accessors, the schema catalog, the
generated instant and identifier, and the column types, and it asks whether a
feature exists at all where one engine has nothing to offer: an insertion-order
column, a write-ahead log to fold, a function that renders an instant. Empty
means the caller falls back or refuses rather than emitting something the engine
will reject.
Some differences turned out not to need a dialect at all. An instant is stored
as text in a fixed layout, so bucketing the log rollup to the hour is a
substring rather than a calendar function, which every engine spells the same
way and an index can be built on. Null-safe equality is IS NOT DISTINCT FROM.
An int64 column is BIGINT. A join with nothing of its own to match still states
a condition. Case-insensitive comparison stays COLLATE NOCASE, because the
schema now asks the engine for the collation that name refers to.
The proof is a Postgres 18 running the data plane: migrations apply, a
collection created over /v1 lands as a table with text, numeric and jsonb
columns and its declared index, and records insert, page, sort, filter across
scalars, multi-value elements and JSON paths, update and delete. A unique index
declared COLLATE NOCASE rejects the second spelling of a handle.
Comparing a JSON sub-path to a number is refused there: the extraction yields
text, the literal binds as a number, and the filter language does not say which
the comparison meant. Compared as text it works.
core.SQLDialect was 345 lines over three files and 22 methods, covering exactly
the differences between SQLite and Postgres: placeholder style, json_extract,
strftime, AUTOINCREMENT, RETURNING, quoting. Nothing has ever called it. Not one
caller in the repo — the only references to the interface were the two files
implementing it.
That matters more than the dead code. BASE_DB_TIER=sql opens a PostgreSQL
connection and is documented as an in-place upgrade, while Base writes
SQLite-flavoured SQL directly: 21 strftime, 6 sqlite_master, 3 json_extract, 3
json_each across the 39 files that build raw queries. Postgres spells all four
differently, so the tier selects a backend the queries cannot address. A file
named dialect.go implementing exactly the missing translation is the most likely
reason that went unread for so long.
hanzoai/orm is already a dependency and its engine package opens by driver name
across postgres, sqlite and mysql, including the ? to $N rewrite this interface
declared and never performed. That is the one to adopt. Keeping a hand-rolled
second copy alongside it — dead, at that — was the thing to remove first.
No deployment sets BASE_DB_TIER, so nothing regresses.
Half of this removal is already on main: 25ae70fcb swept up the staged deletion
of core/bases.go and core/bases_test.go, but not the unstaged edits that took
out what referenced them — so main currently declares Bases() *Bases over a
deleted type and does not compile. This is the other half.
What went, and why it could go: core.Bases was a second, complete implementation
of the product's central noun — a per-org database registry with its own
encryption, lazy open, and concurrent/nonconcurrent pools, reachable through
three core.App methods and gated behind MULTI_BASE/MASTER_KEY. It lived in the
same directory as the live one and disagreed with it on everything else:
data.db + auxiliary.db against org.db, one shared master key against a per-org
DEK from KMS, validateOrgID against validateSlug. Two Bases for one org, neither
aware of the other, had both ever run.
Neither ever did. Its only callers were its own tests, no deployment in the
fleet sets either env var, and nothing outside this repo calls Bases(). The live
path is plugins/org, which provisions on the first request carrying an org and
derives that org's key from KMS.
Also fixes TestNotifyWatcher_CollectionsUpdate, which failed under parallel load
and had been widened once before ("1-2 reloads") without the cause being found.
The cause is that reload count tracks the gaps between writes, gaps track
machine load, and the ceiling was a constant — so a loaded runner pushes a
healthy watcher past any fixed number, and no burst size fixes it because the
burst's duration scales with load too. The ceiling now comes from the burst's
own measured duration divided by the watcher's own debounce window, which takes
machine speed out of the assertion entirely while still catching a watcher that
has stopped debouncing (it emits one reload per change at any speed). The 50ms
window was written twice, once as a literal and once in English in a comment; it
is named once now, and the test reads that name through export_test.go rather
than restating the number.
Verified: full core suite green three times under six competing builds, the
configuration that reproduced the failure.
/v1/bases answered [] for a caller in five orgs, and /v1/bases/{org} answered 403
for every one of them — including their own. It read like an empty account.
resolveJWKSToken mirrors a verified token onto e.Auth, and from then on
requireAuth takes the e.Auth path and builds its IAMUser from the record: id,
email, name. Not the memberships, because a record holds fields and membership
is not one of Base's. So OrgIDs was empty, and both handlers do exactly what they
should with an empty membership set — list nothing, refuse everything.
The claims are already decoded to answer the superuser question, so the org
slugs ride the request beside the other resolved claims and requireAuth reads
them there. One decode, one source, and the answer cannot disagree with the
token the request actually arrived on.
Verified with a real hanzo.id token: /v1/bases lists all five orgs with hanzo
exists:true, /v1/bases/hanzo returns it, and /v1/bases/acme — an org the caller
does not belong to — is still 403. apis and plugins/org pass.
Co-authored-by: Hanzo Dev <[email protected]>
The verification and email-change templates were the last of local auth. Their
endpoints went in the IAM-native rip — bindRecordAuthApi keeps auth-methods and
auth-refresh and nothing else — so both answered 404 for every deployment, not
only IAM ones, and their buttons addressed a route in an admin SPA that no
longer exists. Re-pathing them last commit made them honest; this removes them,
along with everything that existed only to serve them.
That turned out to be a lot, and each piece was already dead on its own terms:
core.EmailTemplate and the two collection fields — no other user
the mails package entire — its two functions were the only callers of its
own template resolver and its own html layout
the $mails JSVM namespace — an empty object once the two binds went, and its
type declarations advertised FIVE helpers, three of which (password reset,
OTP, auth alert) were removed in that same rip and never bound at all, so
hook authors have been reading a contract for functions that do not exist
OnMailerRecordVerificationSend / OnMailerRecordEmailChangeSend — nothing has
been able to fire either since the mails package was their only caller
core.MailerRecordEvent — the type those two hooks carried
Nothing loses a capability. $app.newMailClient().send() is the general sender,
and it is what an app binding its own confirm route would have used anyway; the
two removed binds were convenience wrappers over it.
POST /v1/settings/test/email stays and gets better. It rendered a template
against a throwaway record, so "can this deployment deliver mail" was answerable
only through a flow that no longer exists. It sends a plain message now — no
template, no collection, no record, no token — which is the question an operator
has after editing SMTP settings.
The collection fields are DEFAULTS applied at creation; a collection stores its
own copy, and an options key the struct no longer declares is ignored on read,
so nothing already deployed changes shape. types.d.ts is regenerated by its own
generator rather than hand-edited (the generator still listed the deleted
package, which is why it had to be fixed first).
The mirror landed reading two claims for one fact, and one of them is not
authority. IAM stamps `owner` with the organization of the APPLICATION a token
was minted through — not the subject's own — so honoring it made platform sudo a
property of the app: everyone who signed in through an admin-org application
arrived holding it, over every org's Base in the process. hanzoai/authz says so
at the field, having already been through this.
So Base stops holding an opinion. It decodes the verified claims into
authz.Claims and asks PlatformSudo() — the estate's published predicate, the one
the gateway mints X-User-IsAdmin from and cloud reads through
principal.IsSuperAdmin. Membership of the reserved admin org, at any position in
`orgs`, and a human: a client_credentials token carries no membership set, which
is how a machine is told from a person, and an admin-org machine holding sudo
over every tenant is the same hole from the other side.
Two definitions of one authority is the actual defect. The narrow spelling
(store.IsSuperAdmin on the home org) denies every operator anchored in a brand
org, which is every operator who also does ordinary work; the wide one admits
apps and machines. Neither is Base's call to make, and now neither is made here —
the issuer publishes the predicate, consumers ask it, and it cannot drift.
Nothing else in base linked IAM, so the module goes with it: 88 lines out of
go.sum, and the predicate arrives through the decision leaf that IAM itself uses
rather than the whole server.
Verified against hanzo.id with a PKCE-minted token: /v1/collections 200 with it,
401 without, and settings, logs, backups, scaffolds, collection create, record
create/update/delete and native page+sort+filter all round-trip. Under test an
admin-org app, an admin-org machine, an admin of an ordinary org, the isAdmin
flag, the built-in signing owner and an org merely named `administrators` are all
refused. apis, core and plugins/org pass.
Co-authored-by: Hanzo Dev <[email protected]>
Each pointed at `/_/#/...` — the old admin mount plus a hash route in the SPA
this fork replaced — and each had been dead since that rewrite. They survived
because nothing about them fails loudly: the server answers an unknown path with
the admin page at 200, so a link to nowhere looks like a link.
The first boot message handed an operator a browser URL at
`/_/#/baseinstal/{token}` and printed the CLI command underneath as a fallback.
There is no `baseinstal` route, so the fallback was the only thing that ever
worked; it is now the instruction. Nothing mints a throwaway superuser token to
put in that URL any more either.
The other two are the default verification and email-change mails, whose buttons
went to `confirm-verification` and `confirm-email-change`. Those endpoints were
deleted in the IAM-native rip — bindRecordAuthApi keeps auth-methods and
auth-refresh and nothing else, and both paths answer 404 for every deployment,
not only IAM ones. So the button in the one mail a person is most likely to
click had nowhere to go. The defaults now link to the app itself and say that
the account is where the address is confirmed. {TOKEN} still substitutes, since
an app can bind its own confirm route from a hook and write its own template —
what a default may not do is name a route on the reader's behalf and be wrong.
These are defaults, applied when a collection is created; a collection stores its
own copy, so nothing already deployed changes. The mails and jsvm suites go on
asserting the old shape against their fixtures, correctly: they test that a
template's placeholders get substituted, whatever the template says.
Also removes the last /_/ from the settings placeholder and from smoke.mjs, which
was serving the bundle at the old mount — a visual check against a layout nothing
runs would pass while the real page broke. LLM.md documented the deleted knob.
Two things wore the name "platform" here, and neither is the PaaS at
platform.hanzo.ai. That collision is why "where are my Bases?" had no answer a
person could act on: one prefix, two products, and the one you reached was
whichever you happened to mean.
A Base is what an org gets — its own collections, its own files, its own key —
so the resource a client addresses is /v1/bases. Not /v1/orgs: the org is IAM's
noun and Base does not get a second one. Everything scoped to an org now hangs
off the Base it belongs to (/v1/bases/{id}/config, /creds, /customers), and
compliance moves to /v1/compliance, a sibling of /v1/iam rather than a child of
Base's own resource. The package is `org`, which seven of its files already
were. Nothing addressed the old prefix — no client, no UI, no manifest — so this
renames a surface rather than breaking one.
The auth proxy is DELETED, not renamed. Its five routes reshaped bodies and
forwarded to /v1/iam/login, /signup, /userinfo and /send-verification-code —
which the opaque /v1/iam/{path...} passthrough beside it already reaches. So
Base carried a hand-written login, signup and one-time-code flow whose only job
was to be a second way to reach IAM. Auth is IAM's, and one way is the whole
point of that rule.
The prefix test that already banned /api/ bans this too, so the name cannot come
back: it reads route LITERALS through go/ast, so a comment explaining the ban is
prose and a route is a route. Verified to fail on the exact regression.
Base's own registry was already org-native in every method — OrgDB(orgID),
HasOrg(orgID), orgDir(orgID) — and only the type names said tenant. So this is
the vocabulary catching up with the code, not a change of behaviour.
TenantRegistry -> Bases app.Bases().OrgDB(org)
TenantConfig -> BasesConfig
MultiTenant -> PerOrg it enables per-org databases; say that
MULTI_TENANT -> MULTI_BASE
core/tenant.go -> core/bases.go
Tenant was the wrong noun twice over: it is the multi-tenancy word, and this
product's unit is the thing a customer creates per org — a Base. Identity here
is org, user, project, and nothing else.
The UI never said it; this was Go-side only. 64 packages green.
One test was collateral and is fixed rather than renamed: TestToHTTPHeader
asserts a pure snake_case-to-Header-Case transform, so rewriting its EXPECTED
value while its input stayed "tenant_id" made it assert something false. The case
is dropped — org_id already covers that transform.
`type alias *Collection` is a defined POINTER type, and converting to it
does not shed the method set the way the value form does. Under
encoding/json/v2 that is the difference between a decode and an infinite
recursion:
* v1's indirect() only asks for an Unmarshaler while the type it holds
has methods. A defined pointer type has none, so v1 walked through to
the struct and decoded fields. The trap was invisible.
* v2 resolves the method arshaler on the ELEMENT — arshal_methods.go
declines to inject on a pointer, "let it be injected on the value
receiver" — finds (*Collection).UnmarshalJSON there and calls it
again. Every call re-enters: 4.1M frames, then the goroutine stack
passes 1GB and the process dies.
The image has built with GOEXPERIMENT=jsonv2 since a7660d7e (v0.39.14),
so every collection a JS migration constructs through
jsvm.structConstructorUnmarshal has killed the boot since then. 0.39.11
predates the flag, which is why the same volume boots clean on it — the
crash is data-independent.
The other three sites in this tree already spell it the value way
(tools/types/json_map.go, tools/auth/auth.go, core/settings_model.go),
one of them under this exact comment. This makes it four.
TestCollectionUnmarshalJSON already covered the shape and would have
caught this, but nothing ever ran the suite under the flag the image
ships with — GOEXPERIMENT lived only in the Dockerfile. Under
CGO_ENABLED=0 GOEXPERIMENT=jsonv2 it stack-overflows before this change
and passes after.
(cherry picked from commit bd26d30cbc)
Co-authored-by: Hanzo Dev <[email protected]>
core/sqlite_math_required.go failed any cgo build not carrying the
sqlite_math_functions tag, by referencing a symbol that was never defined
anywhere. So `go build ./...` — no tags, default CGO_ENABLED — did not
work in this repo or in any repo importing core, and the error named a
symbol rather than the problem:
undefined: cgoBuildNeedsSQLiteMathFunctions
The capability it was protecting is real. tools/search emits acos, cos,
sin and radians for geoDistance, and SQLite only has those when compiled
with SQLITE_ENABLE_MATH_FUNCTIONS. Measured, per config:
CGO_ENABLED=0 present (what the Dockerfile ships)
CGO_ENABLED=1 ABSENT <- the one deficient config
CGO_ENABLED=1 -tags sqlite_math_functions present
CGO_ENABLED=1 -tags libsqlite3 present <- blocked anyway
The tag was a proxy for the capability, and the proxy was wrong. Row four
is the config the driver's own doc comment calls production: that tag
links the system sqlcipher, so the amalgamation's -D flag is never
compiled and the tag decides nothing, yet the guard rejected the build.
It also could not have caught a system libsqlcipher genuinely missing the
functions, and no library can satisfy it from the inside, because build
tags are chosen by the top-level build command.
Replace it with a measurement. VerifySQLiteMathFunctions runs the
geoDistance expression once per process against a throwaway in-memory
database and DefaultDBConnect refuses to hand back a connection when the
answer is no, so a deficient binary dies at startup with the three ways
to fix it rather than at a customer's failed search. A probe that could
not run at all is not an error: this reports a measured absence, never a
failure to measure.
Detection is strictly better than before. tools/search already had
TestTokenFunctionsGeoDistanceExec executing the real emitted SQL, and it
fails on the deficient config today; core now carries its own, since
tools/search does not import core and its failure said nothing about
this package.
build, all four configs above exit 0 (was: 2 of 4 refused)
go test ./... CGO_ENABLED=0 64 packages ok, 0 failures
go test ./core -run SQLiteMath CGO=1 FAIL, as it should
Co-authored-by: Hanzo Dev <[email protected]>
zeekay
