mirror of
https://github.com/hanzoai/chat.git
synced 2026-09-30 20:51:17 +00:00
9351e806eb
* feat(data-schemas): pure Mongo-shaped query/update engine for SQLite store
Correctness core of the mongoose->SQLite migration seam: pure, I/O-free
matchesFilter / applyUpdate / projectDoc / sortDocs implementing the exact
Mongo operator subset the chat data methods use ($eq $ne $in $nin $gt/$gte
$lt/$lte $exists $regex $not $and $or $nor; $set $unset $setOnInsert $inc
$push $pull $addToSet). Date-aware comparison, mongo null/absent semantics.
21 unit tests green.
* feat(data-schemas): SQLite DocModel + handle backing the Model API on node:sqlite
DocModel presents the Mongoose Model-API subset the chat data methods use
(findOne/find/findOneAndUpdate/updateOne/updateMany/deleteMany/deleteOne/
countDocuments/distinct/create/insertMany/bulkWrite + chainable QueryBuilder
with select/sort/limit/skip/lean/deleteMany). Docs stored as JSON via node:sqlite
(stdlib, zero new dep); JSON1 expression indexes on unique/anchor fields; exact
mongo semantics delegated to the pure engine; date rehydration on read; no
mongoose, no tenant middleware (Conversation/Message are not tenant-plugged
upstream). createSqliteHandle() returns a mongoose-shaped handle the unchanged
method factories run against. 7 adapter tests green.
* feat(data-schemas): prove conversations+messages fully off mongoose on SQLite
- Decouple message.ts + conversation.ts from the mongoose package: factories now
take a structural DataHandle ({models}) satisfied by BOTH mongoose and the
SQLite handle. Only type-only imports from 'mongoose' remain; zero runtime
mongoose in the data path for this domain.
- engine: type-aware operand coercion so cursor pagination (String(Date) operand
vs Date field) compares chronologically, mirroring mongoose schema casting.
- convoMessage.sqlite.spec: 14 tests running the REAL createMessageMethods /
createConversationMethods against createSqliteHandle — save/upsert, get,
update, delete, deleteMessagesSince, cursor pagination, bulk, archived +
retention-visibility filtering, getConvosQueried, cross-collection deleteConvos,
deleteNullOrEmptyConversations, searchConversation. All green.
Isolates one unrelated pre-existing fork gap via jest mock: this tree's
librechat-data-provider never synced the RetentionMode enum (upstream #13049),
so message.ts/conversation.ts reference an undefined export on the mongoose path
too. Documented for separate fix.
42/42 store tests green (engine 21 + DocModel 7 + contract 14).
* feat(data-schemas): registry-aware createModels + store exports (the seam)
createModels now applies per-domain backend selection via CHAT_STORE_SQLITE
(CSV of collection names -> SQLite DocModel). Unset default = pure mongoose,
live path unchanged; only collections with a CollectionSpec are overridable
(fails closed otherwise). Exports createSqliteHandle/DocModel/CollectionSpec/
DataHandle from the package index. node:sqlite + node:crypto added to rollup
externals. Package builds clean (rollup, EXIT=0); 45/45 store+registry tests green.
* chore(data-schemas): drop local build artifacts + shared-deps symlink from branch
Revert dist/* to base (CI rebuilds bundles deterministically — no local builds)
and untrack the packages/data-schemas/node_modules dev symlink. Source-only branch.
* feat(data-schemas): Batch 2 — Preset, ConversationTag, SharedLink on SQLite
Migrate three self-contained, non-tenant-plugged chat-document domains onto the
DocModel store behind the same seam. Decouple their factories to DataHandle;
createMethods now passes a registry-aware handle to createShareMethods so the
CHAT_STORE_SQLITE flag flips Share (pattern-2) in the live path too.
DocModel extensions (all reusable for later batches):
- compound unique indexes (ConversationTag {tag,user})
- ObjectId-ref casting on write + cross-collection .populate() (SharedLink.messages -> Message)
- findByIdAndUpdate / findOneAndDelete; findOneAndUpdate is now a chainable
QueryBuilder (mutate mode) so .lean()/.select()/.populate() chain after a write
- schema defaults on insert (SharedLink.isPublic:true)
engine: mixed-update semantics (top-level fields fold into $set) + $pullAll.
Contract specs run the REAL createPreset/ConversationTag/Share methods against
createSqliteHandle. 60/60 store+contract+registry tests green; rollup build EXIT=0.
* feat(data-schemas): Batch 3 — Project on SQLite + realtime-vs-storage finding
Project migrated (pattern-1, registry-aware): add DocModel.findById; CollectionSpec
with array defaults; contract spec mirrors api/models/Project.js exact ops
(getProjectByName upsert, $addToSet $each, $pull $in, updateMany $pull).
Prompt/PromptGroup deferred with rationale: they construct mongoose.Types.ObjectId,
use an aggregate $lookup/$unwind pipeline + populate + manual tenant/ACL
(accessibleIds: ObjectId[]) — need an aggregate primitive, not the mechanical
recipe. Categories read-path is already mongoose-free (getCategories is static).
REALTIME finding (verified codebase-wide): chat has ZERO Mongo change-streams /
tailable cursors / .watch(). Its realtime is SSE token streaming (sendEvent /
agent GenerationJobManager) tied to the generation request — application layer,
DB-independent. So NO migrated domain needs a DB-subscription replacement; plain
node:sqlite is correct for all. Hanzo Base realtime is reserved for future
DB-driven push (multi-device live sync, presence, collab sessions) — none today.
Documented at the storage-decision source (collections.ts).
63/63 store+contract+registry green; rollup build EXIT=0.
* feat(data-schemas): Batch 4 — File, Key, PluginAuth, Banner on SQLite
Migrate four non-tenant storage domains onto the DocModel store. Decouple their
factories to DataHandle; createMethods passes the registry-aware handle to
File/Key/PluginAuth (pattern-2); Banner is pattern-1 (~/db/models).
- Key: encrypted roundtrip proven (updateUserKey encrypt -> getUserKey decrypt)
against SQLite; CREDS_KEY/IV set via jest setupFiles (runs before module load).
- PluginAuth: replaced a redundant `new Model().save()` else-branch with
Model.create (equivalent on mongoose, part of the shared Model API — DocModel
supports it). No behavior change.
- Realtime directive recorded: Conversation+Message are the Base-realtime cutover
targets ("Base for realtime, SQLite for storage"); all other migrated domains
are pure storage. Empirically chat has zero Mongo change-streams (realtime=SSE),
so the SQLite store is correct in the interim; Base swap is backend-only.
MCPServer deferred (constructs mongoose.Types.ObjectId + _id ObjectId cursor).
70/70 store+contract+registry tests green; rollup build EXIT=0.
* feat(data-schemas): Batch 5 — tenant-aware DocModel variant + Config on SQLite
Add tenant isolation to the store, mirroring the mongoose applyTenantIsolation
plugin, gated by CollectionSpec.tenantIsolated (unlocks Config/Skill/SkillFile/
SystemGrant):
- scopeFilter: every read/write filter scoped to getTenantId() (SYSTEM bypasses;
no-tenant + TENANT_ISOLATION_STRICT=true fails closed) — wired at the single
candidates() chokepoint.
- stampTenant: inserts stamped with tenantId (create/insertMany/upsert).
- sanitizeTenantUpdate: update payloads cannot mutate tenantId (throws cross-tenant;
strips from $set/$setOnInsert/$unset/top-level) — wired into
mutateOne/updateOne/updateMany/bulkWrite.
Config migrated (compound unique {principalType,principalId,tenantId}); decouple
createConfigMethods to DataHandle; QueryBuilder.session() no-op (single connection).
Fix: anchorWhere binds booleans as 1/0 (node:sqlite rejects JS booleans; json_extract
returns 1/0) — hardens every boolean-filtered collection.
batch5 contract spec proves cross-tenant isolation with REAL createConfigMethods:
per-tenant stamping, identical principals isolated across tenants, tenant-scoped
list/find, no cross-tenant delete. 74/74 green; rollup build EXIT=0.
* feat(data-schemas): Batch 6 — SystemGrant on SQLite + ObjectId read-coercion
SystemGrant migrated (tenant-plugged, same recipe as Config): CollectionSpec with
compound unique {principalType,principalId,capability,tenantId} + tenantIsolated;
add DocModel.exists(); decouple createSystemGrantMethods to DataHandle.
Add ObjectId read-coercion to the engine (coerceId): filters carrying real
mongoose ObjectId operands (SystemGrant.normalizePrincipalId casts USER ids to
Types.ObjectId) now compare against the hex strings the store persists (docs
stringify ObjectIds to hex; _ids are ObjectId-hex). Applied in comparable /
valueEquals / anchorWhere. This is the shared primitive that unblocks the
ObjectId-coupled domains (Skill/SkillFile, MCPServer, Prompt/PromptGroup).
Contract spec runs REAL createSystemGrantMethods with USER principals
(ObjectId path exercised end-to-end): grant/has(exists)/revoke, idempotent upsert,
platform-vs-tenant isolation. 77/77 green; rollup build EXIT=0.
* feat(data-schemas): Batch 7a — ObjectId shim + MCPServer, Skill, SkillFile
Add handle.Types.ObjectId shim (thin: store _ids are already ObjectId-hex,
coerceId resolves comparison/storage): a 24-hex ObjectId class with toHexString/
toString/toJSON/equals/isValid, exposed as SqliteHandle.Types and typed on
DataHandle.Types. createMethods' dbHandle now carries mongoose.Types (real
ObjectIds coerce too).
- MCPServer (pattern-2): full real-method spec — create / findByServerName /
findByObjectId (findById + ObjectId operand) / byAuthor / update / delete +
unique serverName. Decouple models + Types to handle; wired via dbHandle.
- Skill/SkillFile (tenant-plugged): decouple models + Types to handle; storage
contract proven — compound unique, ACL _id-in-accessibleIds ObjectId filtering
via coercion, SkillFile upsert. Their 800-line ACL/validation method layer
rides on these ops (full harness = follow-up).
81/81 green across 10 suites; rollup build EXIT=0.
* feat(data-schemas): Batch 7b — aggregate primitive + Prompt, PromptGroup
Add DocModel.aggregate supporting the bounded stage set the chat methods use
($match / $lookup / $unwind / $sort / $limit / $project), run in JS over
candidate docs; $lookup resolves the mongo collection name to a sibling model
(prompts -> Prompt) and joins via the shared engine. Fix deepCoerceIds: coerce
ObjectId-like values to hex BEFORE structuredClone in create/insertOne (clone
was stripping the shim's methods, serializing productionId as an object husk).
Prompt / PromptGroup migrated (pattern-1): decouple models + Types to handle;
CollectionSpecs with productionId/prompts refs. Contract spec proves the
aggregate primitive directly AND the REAL getPromptGroup end-to-end (casts _id
-> ObjectId, $lookup productionId -> Prompt, $unwind preserveNullAndEmptyArrays).
All storage-tier domains are now on the DocModel. 85/85 green across 11 suites;
rollup build EXIT=0.
* feat(data-schemas): Skill/SkillFile full-method harness + findOneAndUpdate(new:false) fix
Close the Skill/SkillFile proof gap: real createSkillMethods with stub ACL deps
(PermissionService injected) — createSkill (validation + uniqueness), getSkillById,
getSkillByName(accessibleIds ObjectId ACL), updateSkill optimistic version bump,
deleteSkill (+ removeAllPermissions), and SkillFile upsert (new-vs-replace) /
getByPath / list. Skill + SkillFile now fully proven, not just storage-proven.
Fix mutateOne: findOneAndUpdate(new:false, upsert:true) now returns null on an
insert (no pre-image) and the old doc on update — mongoose semantics that
upsertSkillFile's atomic new-vs-replace fileCount detection depends on. Locked
with a direct DocModel.spec test. All prior findOneAndUpdate upserts use new:true
(unaffected).
89/89 green across 12 suites; rollup build EXIT=0.
* feat(data-schemas): Batch 8a — 7 chat-native domains on SQLite
Migrate MemoryEntry, ToolCall, Assistant, Action, AccessRole, Role, AgentApiKey
onto the DocModel store (mechanical: no external subsystem owns them — cloud
/v1/agents is additive, there is no /v1/memory, and app-domain authz has no IAM
equivalent). Decouple all 7 factories to DataHandle; createMethods passes the
registry-aware handle to Role/Memory/AgentApiKey/AccessRole (pattern-2).
Adapt role.initializeRoles off the `new Role().save()` mongoose-document pattern
to the shared Model API (findOne/create/updateOne) — equivalent on both backends.
Contract spec runs the REAL methods: memory set/create(dup-throws)/list/delete,
toolcall create/get/byConvo/delete, assistant/action upsert+get+delete, accessRole
create/find/list/delete, role initializeRoles seeds + listRoles, agentApiKey
create/validate(hash)/list. 96/96 SQLite-spec tests green; rollup build EXIT=0.
(Pre-existing mongoose specs fail identically on base — fork gaps, out of scope.)
* feat(data-schemas): Batch 8b — Agent, AgentCategory, AclEntry, Group on SQLite
Migrate the ObjectId/aggregate/ACL chat-native domains. Decouple factories to
DataHandle (models + Types); wire AgentCategory/AclEntry/UserGroup through the
registry handle. Agent uses the ObjectId shim (version tracking, _id cursor);
AgentCategory uses the aggregate primitive; AclEntry uses bitwise permission
queries; Group links members by id.
Store primitives added/fixed (all reusable):
- aggregate $group (with $sum/$first/$last/$max/$min/$push/$addToSet) — powers
AgentCategory.getCategoriesWithCounts.
- bitwise operators $bitsAllSet/$bitsAnySet/$bitsAllClear/$bitsAnyClear — powers
AclEntry.hasPermission.
- applyUpdate deepCoerceIds the upsert seed (ObjectId shims survived to storage
as hex, fixing findOneAndUpdate-upsert with ObjectId filter fields).
- array-safe index anchor: json_each(...) EXISTS replaces json_extract= so
{field: value} over an ARRAY field (Mongo array-contains, e.g. Group.memberIds,
tags, projectIds) hits the index instead of being excluded by the prefilter.
Contract spec runs REAL methods: agent create/get/update/delete, category $group
counts, acl grant/has(bitwise)/revoke, group create/find/addMember/getUserGroups.
100/100 SQLite-spec tests green across 14 suites; rollup build EXIT=0.
* feat(billing): chat→commerce debit wiring behind COMMERCE_WRITES (default OFF)
Step 2 of the money migration. Default OFF = current local-Mongo billing path
runs byte-for-byte; the Commerce-first fail-closed READ gate is unchanged.
CommerceClient (the previously-dead write methods, now correct):
- recordUsage now debits the billing SUBJECT (billingSubject(owner,email)) — not
the Mongo user id — with amountMicros (lossless micro-USD; commerce rounds to
nearest cent + records exact micros) + requestId (stable per-spend idempotency
key → no double-debit) + totalTokens/provider.
- _flushUsageQueue now passes X-Hanzo-Org per entry (was omitted → debits hit the
wrong tenant and never netted the balance the gate reads). Fixed.
- add deposit() for credits (POST /v1/billing/deposit).
Wiring (flag-gated, additive, fail-open):
- commerceWrites.js: COMMERCE_WRITES gate + recordCommerceDebit (never throws
into the spend path; local Mongo authoritative until cutover).
- createTransaction: after the local debit, ALSO record to commerce when the
flag is ON and the request threaded `subject` (tokenValue is micro-USD;
transaction _id is the idempotency key). Inert until subject is threaded + the
flag flipped.
Tests (executed, PASS): CommerceClient.spec — recordUsage builds the right body
(subject/amountMicros/requestId/totalTokens) + X-Hanzo-Org header, no _namespace
leak; the flag gate is OFF by default.
REMAINING (gated): thread `subject` into txMetadata at the spendTokens call
sites; wire credits (deposit/grantStarter); Step-3 live-verify; then retire local
writes behind the flag. Local Balance/Transaction writes are NOT retired.
* fix(billing): revert chat→commerce debit — the gateway is the single debit authority
The COMMERCE_WRITES chat-debit hook (dc9edbed09, today) is a double-debit
footgun and is reverted. It contradicts the established, LIVE architecture:
- hanzoCloudKey.ts (module doc): every IAM user has one hk- Cloud key; "the
cloud gateway (api.hanzo.ai) debits that key's org commerce balance and
returns 402 when the org runs out. Forwarding the right per-user key ===
correct per-user billing automatically." initialize.ts forwards that key on
every authed request (baseURL api.hanzo.ai/v1).
- packages/api usage.ts (NOTE, 2026-06-27): a prior chat→commerce write in
recordCollectedUsage was a SECOND debit to a mis-keyed account; it was
REMOVED. "Chat must NOT also record usage to Commerce."
- prod CR universe/.../crs/chat.yaml: HANZO_PER_USER_KEY=true, balance gate
off, and in its own words "chat records NO usage to commerce (that write was
removed); the single debit is still cloud's, per the user's hk- key."
Two writers to one ledger for one spend = double charge. The gateway is the ONE
debit authority for AI spend across every product (chat/code/agents/API); a
per-client debit path is the wrong DRY seam. So chat stays a READER of Commerce.
Removed (the whole unused chat→commerce WRITE surface):
- api/models/commerceWrites.js (the COMMERCE_WRITES gate + recordCommerceDebit)
- the recordCommerceDebit call + import in Transaction.js (now a comment
stating the single-debit invariant)
- CommerceClient.recordUsage (the debit) + its now-dead _usageQueue /
_flushUsageQueue machinery + interval; deposit/grantStarter (unused credit
helpers — the real first-chat grant is resolveHanzoCloudKey's direct POST
/v1/billing/grant-starter in packages/api, the correct layer)
- CommerceClient.spec.js (only tested the removed surface)
Kept (CommerceClient is now purely read-only, one responsibility):
checkBalance (fail-closed money gate), getTierConfig, isModelAllowed,
getCreditBreakdown — all live (balanceMethods.js, Balance.js).
Verify: node --check clean on both edited files; zero dangling references to the
removed symbols repo-wide; balanceMethods.spec mocks only the kept READ surface.
Money domain off-Mongo status after this: Commerce (via the gateway) is the
balance/debit authority — no Mongo Balance/Transaction WRITE is authoritative in
prod (gate off). The local Transaction doc remains an in-app usage log only;
retiring/relocating it to the SQLite store is tracked with the cutover.
* feat(identity): User is a thin IAM projection — drop the local password credential (step a)
Identity migration step (a): IAM (hanzo.id) owns identity; chat's User doc is a
thin OIDC-keyed projection (provider='openid', openidId=userinfo.sub, org from
userinfo), never a second credential store.
Confirmed no OIDC path writes a password: openidStrategy.js (508-523) and
process.js createSocialUser build the User with NO password; createUser
(packages/data-schemas) writes no default; the schema field was select:false.
The ONLY writers were two local-email flows, both gated OFF in prod
(ALLOW_REGISTRATION=false, ALLOW_EMAIL_LOGIN=false).
Changes (source only — Docker rebuilds packages/data-schemas dist via
`pnpm run frontend`):
- schema/user.ts: remove the `password` field. The User projection carries no
local secret. (IUser keeps `password?: string` — always undefined — so the
disabled local strategy / comparePassword still compile; the full
local-strategy teardown lands with the cutover.)
- AuthService.registerUser: stop writing password (local signup stores no
credential).
- AuthService.resetPassword: reject — "Password reset is managed by Hanzo IAM
(hanzo.id)." Nothing to reset locally.
Login-safety: the authed OIDC flow (openidStrategy) references `password` ZERO
times — provably untouched by this change. comparePassword/localStrategy run
only when local login is enabled (off in prod), so removing the field is inert
for the live path.
Verify: data-schemas builds clean; store-registry + convo/message + user-schema
specs green (17/17 targeted); full suite identical to clean tree
(1210 pass / 250 pre-existing mongo-env fails / 100 skip — my change adds zero
failures). LIVE Dave-login verification is the deploy-time gate in the cutover
runbook — this branch is not deployed, so prod is untouched (default path stays
pure-mongoose + password field until the cutover deploy).
STOP before step (b): OPENID_REUSE_TOKENS is the documented login-breaker
(chat.yaml: =true → /api/auth/refresh 403 at hanzo.id → login dead). Do NOT flip
until the IAM refresh-token fix lands and is live-verified.
---------
Co-authored-by: Hanzo Dev <[email protected]>
6 lines
360 B
JavaScript
6 lines
360 B
JavaScript
// Jest setupFiles: runs before any test-module import, so crypto/index.ts reads
|
|
// these at module-load time. Only sets test defaults when unset (never clobbers
|
|
// a value a specific test provides). AES-256: 64-hex key, 32-hex iv.
|
|
process.env.CREDS_KEY = process.env.CREDS_KEY || 'f'.repeat(64);
|
|
process.env.CREDS_IV = process.env.CREDS_IV || '0'.repeat(32);
|