Files
zeekay b29f60c059 Answer every URL Google knows at one address
Search Console had 5,812 hanzo.ai URLs as 404, 1,323 as duplicates
without a canonical and 165 as noindex. Three causes: the export shipped
a noindex page that forwarded in the browser at each retired address,
every page also answered at /x/, /x.html and www, and each page's .md
twin was an indexable copy of it.

Retired addresses are now 301s at the edge, rendered into hanzoai/universe
by scripts/edge.mjs from lib/edge.ts: MOVED grows the old product
taxonomy, the first site's routes, pages later dropped and the gateway's
old docs; SHELVES sends the rest of /docs/ to docs.hanzo.ai; and
lib/data/retired.json holds the 256 model pages the catalog has dropped,
read against the catalog each build renders. The shells go, except the
four that are Go import paths, which ?go-get=1 still reads.

The sitemap lists only pages at their own address, each with a lastmod
from its files' last commit, so deploy and gates check out the whole
history.
/privacy, /terms and /cookies stay up for the shared footer but point at
/legal/*, so they leave the sitemap. /pay and /v1 join the Disallow list.

Gates run the edge's rules over the export: every retired address lands
on a page in one hop, every page answers at one address, every sitemap
URL is final and dated, and no page links a redirect. scripts/crawl.mjs
asks production the same questions, and reads a Search Console export.
2026-09-29 21:34:44 -07:00

101 lines
3.7 KiB
JavaScript

/**
* The site's own pages, as the ⌘K palette's index.
*
* A palette that indexes only products answers "no results" about /pricing —
* a page the header links three inches above it. So the pages go in, and they
* are DERIVED rather than listed: this walks the App Router tree, keeps what
* `lib/publish` publishes, and writes the result. A hand-written list of "pages
* worth finding" is a second opinion about what the site is, and it goes stale
* the first time someone adds a route.
*
* TOP LEVEL only. Everything deeper is the cloud taxonomy — /cloud/<slug> and
* /products/<id> — which the palette already indexes from `catalog.json` with
* the names commerce gives them. Walking those here would put every primitive
* in the list twice, once under the name it is sold as and once under a
* title-cased URL slug.
*
* It imports the TypeScript sources directly, which Node strips types from on
* its own from 22.18 — the floor `.nvmrc` and `engines` already name, and the
* same call `scripts/noindex.mjs` makes. The alternative is a second copy of
* the policy in JavaScript, which is what these scripts exist to prevent.
*/
import { writeFileSync } from 'node:fs'
import { join } from 'node:path'
import { MOVED, policy } from '../lib/publish.ts'
import { routes } from '../lib/routes.ts'
const ROOT = join(import.meta.dirname, '..')
const OUT = join(ROOT, 'lib', 'data', 'pages.json')
/**
* How a slug is spelled when title-casing gets it wrong.
*
* An ORTHOGRAPHY table, not a taxonomy: it says how to write a word this site
* already uses, and it can neither add a page nor withhold one. "Api" is simply
* a misspelling of API, and no walk can know that.
*/
const SPELLING = {
ai: 'AI',
api: 'API',
authz: 'Authorization',
captable: 'Cap table',
cli: 'CLI',
dns: 'DNS',
docdb: 'DocDB',
docs: 'Documentation',
faq: 'FAQ',
gpu: 'GPUs',
gui: 'GUI',
hsm: 'HSM',
iam: 'IAM',
ide: 'IDE',
idv: 'Identity verification',
in: 'in',
kms: 'KMS',
kv: 'KV',
llm: 'LLM',
llms: 'LLMs',
mcp: 'MCP',
mq: 'MQ',
o11y: 'Observability',
oss: 'Open source',
sdk: 'SDKs',
sdks: 'SDKs',
sms: 'SMS',
sql: 'SQL',
ui: 'UI',
}
const label = (slug) =>
SPELLING[slug] ??
slug
.split('-')
.map((word) => SPELLING[word] ?? word[0].toUpperCase() + word.slice(1))
.join(' ')
const pages = routes(join(ROOT, 'app'))
.map(({ path }) => path)
// WHAT THIS SITE HAS, not what a crawler may list. Those are two questions and
// this asked the second one, so the seven ROOMS — /chat, /dev, /board, /work,
// /drive, /inbox, /tasks — were missing from the list the chrome navigates by.
//
// They are `noindex` for a reason that says nothing about people: a room
// exports as an empty shell because everything in it arrives from the session
// after hydration, so a crawler is offered nothing. `sitemap.ts` asks
// `indexable()` and still withholds them, which is where that decision
// belongs and where it stays.
//
// What it cost: the header's Chat door is kept only where the site publishes
// the page behind it, so ⌘K lost the one address most readers open it to
// find. A door the product is named after, absent from the menu whose whole
// job is to name the doors.
.filter((path) => path !== '/' && path.indexOf('/', 1) < 0 && policy(path) !== 'private' && !MOVED[path])
.map((path) => ({ id: path.slice(1), title: label(path.slice(1)), href: path }))
if (pages.length === 0) {
throw new Error('no published top-level routes: app/ is the tree this reads')
}
writeFileSync(OUT, JSON.stringify(pages, null, 2) + '\n')
console.log(`pages: ${pages.length} published top-level routes -> lib/data/pages.json`)