Files
hanzo.ai/lib/publish.ts
zeekay 70349b34c7 beta: a post about one topic, with that topic's page and card
The share card offers Hanzo, the products and the betas the member asked
for, the rest under More by group. A topic picks the posts Shuffle offers,
the link (/beta/for/<topic>, tagged utm_content) and the card it unfurls
to: that page carries the topic's title, line and card, a canonical naming
itself and the one noindex (lib/publish SHARES), and sends a person on to
/beta with the query kept, so ref and the campaign land as before.

lib/hanzo/topics is the list, its words the site's and BETAS's own.
generate-og draws a card per topic from it, palette-quantized with sharp:
49 cards, 2.9 MB. The cards gate reads every card's rendered words.

Claude-Session: https://claude.ai/code/session_017vrVus6PPBgP5YRzKV1iqW
2026-10-26 12:03:39 -07:00

594 lines
25 KiB
TypeScript

/**
* Whether a crawler may FETCH a route, and whether it may INDEX it.
*
* Two questions, not one, and the controls that answer them are in tension. A
* `Disallow` in robots.txt stops the FETCH, and a page a crawler never fetches
* is a page whose `noindex` it never reads. Google states it outright — "For
* the noindex rule to be effective, the page or resource must not be blocked
* by a robots.txt file" — so applying both to one route leaves only the weaker
* of the two in force, and the bare URL stays listable from any inbound link.
*
* So there are two answers a route can have besides `public`, and it has
* exactly one of them:
*
* private the crawler must never fetch it. `Disallow` in robots.txt, and
* absent from sitemap.xml, because offering what you forbid is a
* contradiction. NO `noindex`: behind a `Disallow` it can never be
* read, and writing one claims a protection it does not have.
*
* noindex the crawler may fetch it and must not index it. Absent from
* sitemap.xml, and `noindex` in the page itself — the only one of
* the three controls that removes a page a crawler already found.
* Deliberately NOT disallowed, because the `Disallow` is exactly
* what would stop the `noindex` being read.
*
* The POLICY is named for the control; the LISTS are named for the reason. Two
* routes can be withheld from the index for entirely different reasons and
* still need the identical mechanism, so the mechanism is written once.
*
* `policy()` is the one predicate. Every control below is a projection of it,
* and no control reads the lists directly.
*/
import rooms from './data/rooms.json' with { type: 'json' }
/**
* Never fetched. A SURFACE: everything beneath it too. The auth surfaces
* (/signup beside /login: it draws the sign-in, which is for a person, not an
* index), the checkout (another export, mounted at /pay), the API calls this origin
* answers for its own pages (/v1), which answer 401 and 405 to a crawler, and
* /forward, hanzo.chat's root, which counts a visit and sends it to /chat.
*/
export const PRIVATE: readonly string[] = ['/auth', '/account', '/login', '/signup', '/forget', '/join', '/onboarding', '/invite', '/pay', '/v1', '/forward']
/**
* The rooms: routes under `app/(app)`, which render a session rather than copy.
*
* Withheld for what they ARE, so this is not a decision and there is nothing to
* approve. Derived by `scripts/sync-rooms.mjs` off the same tree that decides
* whether the room exists at all — the four newest rooms shipped in sitemap.xml
* because the list used to be typed here, and one of them was thin enough to
* take the two content gates down with it. A SURFACE.
*/
export const ROOMS: readonly string[] = rooms
/**
* Routes whose copy no owner has approved. They build, they answer on their own
* URL, they are fetched, and they are indexed nowhere. A SURFACE.
*
* Deleting a route from this list is the approval — there is no second switch,
* and no page can approve itself.
*/
export const UNAPPROVED: readonly string[] = [
// `/tasks` is a marketing page under `app/(marketing)`, not one of the rooms
// — the room that shares its name is `/board`. Its copy is unwritten, which is
// the only reason anything belongs on this list.
'/tasks',
// `/university/portal` draws a sample student's terminal, grades and certificate
// from fixed data: a preview of the product, not a page a crawler should index.
'/university/portal',
// EMPTIED — owner approval, 2026-08-13. All 78 routes that were
// withheld here are now published: they answer on their own URL, they are in
// sitemap.xml, they carry no `noindex`, and the chrome links them.
//
// The list is kept rather than deleted because it is the MECHANISM, not the
// decision. It is how a route is withheld while its copy is written, and the
// gate below still holds whatever is in it in both directions — so the next
// unwritten page has somewhere to go, and does not need this file rebuilt.
//
// Withholding a route from the index also withholds it from the MENU: the
// header calls `policy()` through `shown()`, so Solutions, Pricing, Learn,
// Research and Support were absent from the nav for exactly this reason, not
// because nobody had added them.
//
// The rooms used to be listed here and are `ROOMS` above: a room is withheld
// for being a room, which is a fact about the tree rather than a judgement
// about its copy, so it is read off the tree and never typed.
]
/**
* Retired addresses, and the page that answers for each now.
*
* The edge answers each with a 301 in one hop, whatever its form (`/sql/`,
* `/sql.html`, www), and keeps the query: `scripts/edge.mjs` renders these as
* the rules hanzoai/universe serves in front of the export. So the export ships
* no page at a retired address, with one exception: a Go import path (lib/
* go-modules), whose page carries the `go-import` tag and is served to
* `?go-get=1`, the one request the edge does not redirect.
*
* A target is a published page or another host, never another forward; the
* gates hold both ends. Retired MODEL pages are the catalog's, not this list's:
* `lib/data/retired.json`, read against the model pages each export writes
* (lib/moved).
*
* The catalog's categories were the product index before the layers were,
* and `/sql`, `/kv`, `/docdb` and `/vector` sold stores the stack does not run:
* an org's data is its own SQLite Base, and analytics is the Datastore.
*/
export const MOVED: Record<string, string> = {
// One page per subject: the SDKs (and the OpenAI and Anthropic shapes the
// gateway answers) are /sdks; the work Hanzo shipped is /customers; the
// account is Settings; the app's rooms are /, /chat and /dev.
'/docs/sdk': '/sdks',
'/customers/proof': '/customers',
'/user-profile': '/settings',
'/organization-profile': '/settings/organization',
// Settings is one tree, /settings, and none of it is Dev's (owner,
// 2026-10-22). Every address once written under /dev/settings — the
// builder's sections and this site's own forwards — answers at the page
// @hanzo/app `settings()` names for its last segment; a section with no
// page of its own is General.
'/dev/settings': '/settings',
'/dev/settings/general': '/settings',
'/dev/settings/account': '/settings',
'/dev/settings/privacy': '/settings',
'/dev/settings/capabilities': '/settings',
'/dev/settings/memory': '/settings',
'/dev/settings/code': '/settings',
'/dev/settings/integrations': '/settings',
'/dev/settings/notifications': '/settings',
'/dev/settings/organization': '/settings/organization',
'/dev/settings/members': '/settings/organization/members',
'/dev/settings/organization/members': '/settings/organization/members',
'/dev/settings/billing': '/settings/organization/billing',
'/dev/settings/organization/billing': '/settings/organization/billing',
'/dev/settings/usage': '/settings/organization/usage',
'/dev/settings/organization/usage': '/settings/organization/usage',
'/dev/settings/keys': '/settings/organization/keys',
'/vibe': '/',
'/beluga': '/',
// The landing the owner read here before it replaced `/` (2026-10-16).
'/home2': '/',
'/products/ai': '/products/models',
'/products/apps': '/products/agents',
'/products/compute': '/products/machines',
'/products/data': '/products/memory',
'/products/dev': '/products/agents',
'/products/infrastructure': '/products/machines',
'/products/network': '/products/machines',
'/products/observe': '/products/record',
// One chat: /chat. The studio product lives at studio.hanzo.ai.
'/ai-studio': '/chat',
'/team/chat': '/chat',
// Hanzo Team is its own site: the product and its AI teammates are
// hanzo.team's pages now, the teammates at /agents/<id>.
'/team': 'https://hanzo.team',
'/team/details': 'https://hanzo.team',
...Object.fromEntries(
['algo', 'art', 'cal', 'core', 'data', 'db', 'des', 'dev', 'fin', 'mark', 'mu', 'opera', 'sec', 'su', 'vi'].map((id) => [
`/team/${id}`,
`https://hanzo.team/agents/${id}`,
]),
),
'/products/science': '/research',
'/products/security': '/products/identity',
'/products/web3': '/products/record',
'/sql': '/products/memory',
'/kv': '/products/memory',
'/docdb': '/products/memory',
'/vector': '/products/memory',
'/database': '/datastore',
// Developers is the docs.
'/developers': 'https://docs.hanzo.ai',
// One page per programme: the open-source royalty is /oss and the partner
// programme is /affiliate. These carried other numbers for the same things.
'/open-source-fund': '/oss',
'/open-source/dividends': '/oss',
'/partners': '/affiliate',
// Zen's page is /zen, beside /enso and /kai.
'/models/zen': '/zen',
// Second copies of /customers and of /products.
'/case-studies': '/customers',
'/overview': '/products',
// The Solutions section before lib/data/solutions.ts: capability pages and
// industry pages two segments deep, each to the solution or product that
// answers for it now.
'/fintech': '/solutions/financial-services',
'/solutions/audio': '/solutions/content-creation',
'/solutions/capabilities': '/solutions',
'/solutions/fine-tuning': '/solutions/reserved',
'/solutions/industries': '/solutions',
'/solutions/industries/enterprise': '/enterprise',
'/solutions/industries/finance': '/solutions/financial-services',
'/solutions/industries/healthcare': '/solutions/healthcare',
'/solutions/industries/retail': '/solutions/retail',
'/solutions/rag': '/solutions/research',
'/solutions/realtime': '/solutions/engineering',
'/solutions/stacks': '/solutions',
'/solutions/stacks/analytics': '/solutions/data',
'/solutions/stacks/devportal': '/solutions/engineering',
'/solutions/stacks/ecommerce': '/solutions/retail',
'/solutions/stacks/saas': '/solutions/engineering',
'/solutions/vision': '/models',
// Second copies of one product each: /ai was an earlier landing for the
// platform, /llm a stub of the one API, /storage a second page for the object
// store /s3 describes.
'/ai': '/platform',
'/llm': '/api',
'/storage': '/s3',
// Unlinked pitches for products the platform does not serve.
...Object.fromEntries(
[
'assets',
'attestations',
'bridge',
'defi-stack',
'graph',
'id',
'ipfs',
'mpc',
'nodes',
'quest',
'safe',
'settlement',
'storage',
'wallet',
'web3-cloud',
].map((slug) => [`/blockchain/${slug}`, '/blockchain']),
),
'/blockchain/dex': '/blockchain/exchange',
'/defi': '/blockchain',
'/defi/exchange': '/blockchain/exchange',
'/defi/staking': '/blockchain',
'/web3': '/blockchain',
// Forwards that were pages here: each said one sentence and sent the reader on.
'/affiliates': '/affiliate',
'/code': '/dev',
'/contact/sales': '/contact-sales',
'/fintech/payments': '/payments',
'/products/integrations': '/integrations',
'/products/platform': '/products/machines',
'/sentry': '/sentinel',
'/soc2': '/trust',
'/status': 'https://status.hanzo.ai',
// The docs are docs.hanzo.ai (SHELVES sends the rest of /docs/ there). These
// are the addresses whose page there has another name.
'/docs': 'https://docs.hanzo.ai',
'/docs/audit': 'https://docs.hanzo.ai/docs/openapi/audit',
'/docs/local': 'https://docs.hanzo.ai/docs/quickstart',
'/docs/orgs': 'https://docs.hanzo.ai/docs/run-your-org',
'/docs/templates': 'https://docs.hanzo.ai/docs/deploy',
// The gateway's docs before docs.hanzo.ai: the proxy it grew from.
'/docs/anthropic_completion': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/pass_through/bedrock': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/pass_through/cohere': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/pass_through/google_ai_studio': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/pass_through/vertex_ai': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/proxy/caching': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/proxy/custom_pricing': 'https://docs.hanzo.ai/docs/pricing',
'/docs/proxy/enterprise': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/proxy/guardrails': 'https://docs.hanzo.ai/docs/projects/hanzoai/gateway',
'/docs/proxy/tag_routing': 'https://docs.hanzo.ai/docs/usage/routing',
'/docs/proxy/team_based_routing': 'https://docs.hanzo.ai/docs/usage/routing',
'/docs/proxy/team_budgets': 'https://docs.hanzo.ai/docs/concepts/budgets',
'/docs/proxy/users': 'https://docs.hanzo.ai/docs/usage/teams',
'/docs/proxy/virtual_keys': 'https://docs.hanzo.ai/docs/api-keys',
'/docs/wildcard_routing': 'https://docs.hanzo.ai/docs/usage/routing',
// The first site, a single-page app whose router answered these.
'/case-study': '/customers',
'/community': '/support',
...Object.fromEntries([...Array(11).keys()].map((n) => [`/case-study/${n}`, '/customers'])),
'/hanzoapp': '/download',
'/hanzobot': '/bot',
'/hanzocode': '/dev',
'/hanzodev': '/dev',
'/help': '/support',
'/index-old': '/',
...Object.fromEntries(['', '2', '3', '4', '5', '6'].map((n) => [`/index${n}`, '/'])),
'/install': '/download',
'/invoices': '/account/invoices',
'/leadership': '/about',
'/nodes': '/node',
'/observability': '/o11y',
'/referral-program': '/affiliate',
'/referrals': '/affiliate',
'/usage': '/account/usage',
'/zen-models': '/zen',
'/zen-of-hanzo': '/philosophy',
// Pages the Next site later dropped.
'/benchmarks/autonomous-companies': '/benchmarks',
'/cloud-site': '/cloud',
'/cloud/api': '/api',
'/cloud/attestations': '/cloud',
'/cloud/cdn': '/edge',
'/cloud/cost': '/pricing',
'/cloud/fine-tuning': '/solutions/reserved',
'/cloud/finetune': '/solutions/reserved',
'/cloud/jobs': '/cloud',
'/cloud/rerank': '/cloud/embeddings',
'/cloud/sdks': '/sdks',
'/solutions/api-development': '/solutions/engineering',
'/solutions/data-engineering': '/solutions/data',
'/solutions/enterprise-ai': '/enterprise',
// Go modules renamed under the one import path each now has.
'/amqp': '/mq',
'/go-openai': '/openai',
'/go-openrouter': '/openrouter',
'/sendgrid-go': '/sendgrid',
// Case studies with no page of their own: the index names them.
...Object.fromEntries(
[
'arca',
'bellabeat',
'casper-blockchain',
'cover-build',
'keek-social',
'myle-tap',
'palladium',
'skully',
'stoned',
'trillerfest',
'unikoin-gold',
].map((id) => [`/customers/${id}`, '/customers']),
),
// The product taxonomy before the layers: /products/<category>/<product>, each
// to the page that sells the product now, or its layer where none does.
...Object.fromEntries(
Object.entries({
apps: {
agent: '/agents',
analytics: '/analytics',
base: '/base',
captable: '/captable',
chat: '/products/agents',
cloud: '/cloud',
commerce: '/commerce',
console: '/console',
dataroom: '/dataroom',
flow: '/flow',
mcp: '/mcp',
operative: '/operative',
sign: '/sign',
},
async: {
auto: '/auto',
cron: '/automations',
flow: '/flow',
mq: '/mq',
pubsub: '/pubsub',
queues: '/mq',
stream: '/stream',
tasks: '/automations',
},
compute: { functions: '/functions', gateway: '/gateway', machines: '/machines', runtime: '/machines' },
cx: { crm: '/products/company', 'cx-suite': '/products/company', erp: '/products/company' },
data: {
datastore: '/datastore',
'datastore-vector': '/datastore',
documentdb: '/products/memory',
kv: '/products/memory',
search: '/search',
sql: '/products/memory',
'sql-vector': '/products/memory',
storage: '/s3',
vector: '/products/memory',
},
growth: {
ads: '/insights',
engage: '/insights',
experiments: '/insights',
insights: '/insights',
'web-analytics': '/analytics',
},
ml: {
enso: '/models/enso',
evaluation: '/products/models',
'feature-store': '/products/models',
jin: '/jin',
kensho: '/research',
mu: '/research',
notebooks: '/products/models',
pipelines: '/products/models',
registry: '/products/models',
satori: '/research',
serving: '/products/models',
training: '/products/models',
tuning: '/solutions/reserved',
zen: '/zen',
},
observability: { logs: '/o11y', metrics: '/metrics', telemetry: '/telemetry', traces: '/o11y' },
platform: {
dns: '/dns',
edge: '/edge',
guard: '/guard',
hke: '/machines',
iam: '/iam',
kms: '/kms',
networking: '/products/machines',
platform: '/platform',
tunnel: '/tunnel',
},
}).flatMap(([category, products]) =>
Object.entries(products).map(([product, to]) => [`/products/${category}/${product}`, to]),
),
),
'/products/async': '/automations',
'/products/cx': '/products/company',
'/products/growth': '/insights',
'/products/ml': '/products/models',
'/products/observability': '/o11y',
}
/**
* Retired shelves: every path under the key moves to the same path under the
* value, one hop, query kept. `keep` names the pages still published under the
* shelf, which the edge serves rather than sends on. An exact MOVED entry wins.
*/
export const SHELVES: readonly { from: string; to: string; keep: readonly string[] }[] = [
{ from: '/docs/', to: 'https://docs.hanzo.ai/docs/', keep: [] },
]
/**
* Routes with nothing to read: a shell that waits for a session, or a retired
* address kept as a page for `go get`.
*
* A sitemap offers pages for INDEXING and these have nothing to index — a
* session shell is a soft 404, and a forward is a redirect, which Google's own
* sitemap guidance says to leave out. Count is not the dimension that matters
* here; content is.
*
* A PAGE, not a surface, and the distinction is load-bearing: a page with
* something to say may live under a retired shelf. Emptiness is a property of a
* page. Privacy is a property of a surface — everything under `/account` is
* private, including the page nobody has written yet.
*
* The way OFF this list is to give the page something to say. The gate holds
* the list to that in both directions: nothing offered to a crawler may read
* shorter than these, and none of these may grow copy without being published.
*/
export const EMPTY: readonly string[] = [
'/referral',
// The beta member's own page: their access, drawn once IAM and billing answer.
'/beta/status',
// The apps in beta that draw inside the app (app/apps): a door to sign in, and
// one page for every /apps/<id> (lib/edge), with nothing for a crawler.
'/apps',
// Retired addresses that are also Go import paths (lib/go-modules): the page
// answers `?go-get=1` with its tag, and the edge sends every other request on.
'/ai',
'/docdb',
'/status',
// The beta's short link: it forwards to /pricing with the beta's coupon.
'/betaa',
]
/**
* Share pages: /beta/for/<topic> (app/beta/for/[topic]), the address a member's
* post links to. A social crawler fetches one for its title and card, and a
* person is sent on to /beta with the link's query, so it is fetched and
* indexed nowhere: what it says, /beta and the product pages say. A SURFACE.
*/
export const SHARES: readonly string[] = ['/beta/for']
/**
* Mail: the bodies of the emails the site sends, kept at an address so a
* message's images load and a reader can open it in a browser. Files under
* `public/email`, not pages of the site: fetched, and indexed nowhere. A SURFACE.
*/
export const MAIL: readonly string[] = ['/email', '/unsubscribe']
/**
* Pages this export ships for another host, which serves them as its own and
* owns their address. Neither private nor withheld: each is indexed where it
* lives, so its canonical names that host, its own URL here forwards there at
* the edge, and this site's sitemap does not offer it.
*
* /app is hanzo.app's root — the page that gets you the Hanzo App (universe
* static-sites `hanzo-app`).
*/
export const HOSTED: Readonly<Record<string, string>> = {
'/app': 'https://hanzo.app',
}
export type Policy = 'public' | 'private' | 'noindex' | 'hosted'
/**
* The route, without the query.
*
* A static export serves the same bytes for `/login` and `/login?next=/account`,
* so they are one page and must have one policy — and a crawler that may fetch
* the second has fetched the first.
*/
const path = (route: string) => route.split(/[?#]/)[0]
/** A route is "under" a surface when it is the surface or a descendant of it. */
const under = (route: string, surface: string) => {
const p = path(route)
return p === surface || p.startsWith(surface + '/')
}
/** The one predicate. Everything else here is a projection of it. */
export function policy(route: string): Policy {
if (PRIVATE.some((surface) => under(route, surface))) return 'private'
if (ROOMS.some((surface) => under(route, surface))) return 'noindex'
if (UNAPPROVED.some((surface) => under(route, surface))) return 'noindex'
if (EMPTY.some((page) => path(route) === page)) return 'noindex'
if (SHARES.some((surface) => under(route, surface))) return 'noindex'
if (MAIL.some((surface) => under(route, surface))) return 'noindex'
if (Object.hasOwn(HOSTED, path(route))) return 'hosted'
// A retired address answers with a 301, so nothing may offer or link it.
if (MOVED[path(route)] || SHELVES.some(({ from, keep }) => path(route).startsWith(from) && !keep.some((k) => under(route, k)))) {
return 'noindex'
}
return 'public'
}
/**
* Public pages whose canonical names another page: the same document at a
* second address that stays up. The legal pages' short addresses stay because
* the shared footer (@hanzogui/shell) links them; /bot/details stays because
* hanzo.bot is rooted at it (deploy.yml), and it draws what /bot draws. They
* answer; the sitemap offers the owner instead.
*/
export const ALTERNATE: Readonly<Record<string, string>> = {
'/aup': '/legal/acceptable-use',
'/bot/details': '/bot',
'/cookies': '/legal/cookies',
'/privacy': '/legal/privacy',
'/terms': '/legal/terms',
}
/** Control 1 — `app/sitemap.ts` offers a crawler only what is public, at its own address. */
export function indexable(route: string): boolean {
return policy(route) === 'public' && !ALTERNATE[path(route)]
}
/**
* One surface, as the robots.txt lines that mean it.
*
* A robots.txt path matches as a STRING prefix and withholding is a PATH
* SEGMENT — two different rules, and writing the segment as if it were the
* string is what put the live /authz product page behind `Disallow: /auth`.
* RFC 9309 §2.2.2 gives the grammar to say what is meant: `$` ends the match.
* So one surface becomes the three URL FORMS that are the same route — itself
* anchored, everything beneath it, and itself with a query on it. The third is
* not decoration: `/login$` does not match `/login?next=/account`, and a login
* URL is exactly the one that gets linked with a query. Together they cover
* `under()` over URLs, not merely over paths.
*
* Unexported, and the only place a Disallow line is spelled. `app/robots.ts`
* emits DISALLOW and cannot reach the lists, so a bare string prefix is not a
* thing that surface can write.
*/
const lines = (surface: string) => [surface + '$', surface + '/', surface + '?']
/** Control 2 — the robots.txt `Disallow` lines. Private routes only. */
export const DISALLOW: readonly string[] = PRIVATE.flatMap(lines)
/**
* Control 3 — the tag a withheld page carries, in its one spelling.
*
* `scripts/noindex.mjs` writes it into the export; no page writes it and no
* page can. It was page `metadata` reading this list, and that made the only
* control a page states about ITSELF the only one a page could forget: a route
* added to a list without the matching line in its `page.tsx` shipped with no
* tag at all, and every gate stayed green. Most of the pages that need it are
* `'use client'` besides, where Next forbids a `metadata` export outright, so
* the per-page spelling would have been twenty `layout.tsx` files carrying one
* line each — and the twenty-first page would not have had it either.
*/
export const NOINDEX = '<meta name="robots" content="noindex, nofollow, nocache"/>'
/**
* Does a robots.txt path pattern match a route, by the crawler's own rule?
*
* RFC 9309 §2.2.2: the pattern matches a PREFIX of the path, `*` stands for any
* sequence of characters, and `$` anchors the end. This is what the gate runs
* every offered route through, so the two rules are compared rather than
* assumed to agree — the assumption is the defect.
*/
export function matches(pattern: string, route: string): boolean {
const anchored = pattern.endsWith('$')
const body = anchored ? pattern.slice(0, -1) : pattern
const source =
'^' +
body
.split('*')
.map((part) => part.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'))
.join('.*') +
(anchored ? '$' : '')
return new RegExp(source).test(route)
}