mirror of
https://github.com/hanzo-apps/hanzo.ai.git
synced 2026-10-29 07:20:43 +00:00
zeekayandHanzo Dev
fafdb890a4
`Disallow: /auth` de-crawled /authz. A robots.txt path matches as a STRING
prefix (RFC 9309 §2.2.2) and withholding is a PATH SEGMENT, so the one bare
prefix took the live Authz product page with it — the same build offered
/authz in sitemap.xml and forbade it in robots.txt. `under()` had the segment
rule and was asserted; nothing asserted that robots.txt agreed.
And the Disallow disabled the noindex. Google: "For the noindex rule to be
effective, the page or resource must not be blocked by a robots.txt file." A
crawler that may not FETCH a page never reads the `noindex` in it, so a route
carrying both keeps only the weaker control while reading as if it had the
stronger — the bare URL stays listable from any inbound link. Control 2 was
switching control 3 off.
Both come from the same place: one WITHHELD list applied to all three
controls. Two different policies were collapsed into one, so each route got
every control whether or not the control could work for it.
Split:
private never fetched. Disallow, and absent from sitemap.xml, because
offering what you forbid is a contradiction. NO noindex — behind
a Disallow it can never be read, and writing one claims a
protection it does not have.
unapproved fetched, never indexed. Absent from sitemap.xml, and noindex in
the page. Deliberately NOT disallowed, because the Disallow is
exactly what would stop the noindex being read.
`policy()` is the one predicate and returns which of the three a route is;
every control is a projection of it and none reads the lists. One prefix
becomes the pair of RFC 9309 patterns that mean it — `/auth$` and `/auth/` —
from one unexported constructor, and `app/robots.ts` emits DISALLOW and cannot
reach the prefixes, so the bare-string spelling is not available to the
surface that got it wrong. WITHHELD is gone.
The gates now check the two files AGAINST EACH OTHER over the whole export
rather than each against the list it came from: no route offered in
sitemap.xml may match any Disallow line under the crawler's own rule
(`matches()`, RFC 9309, `*` and `$`), and no route may be both disallowed and
noindexed. The noindex assertion iterates the list instead of naming
risk.html, because the third control is the one written per page: a route
added to UNAPPROVED whose author forgets the metadata line shipped with no
noindex at all and every gate stayed green.
hanzo.yml: the lint step is gone. Steps run in order and it cannot pass —
`next lint` was removed in Next 16, and eslint direct fails because
typescript-eslint 8.65 rejects TypeScript 7.0.2 — so it was not a weak gate
but a wall in front of the real one. Types stay checked by the build inside
`gates`. deploy.yml records that its `push` trigger is inert for the pushes
this estate makes: merge by pull request or the gates do not run.
Co-authored-by: Hanzo Dev <[email protected]>
23 lines
918 B
TypeScript
23 lines
918 B
TypeScript
import type { MetadataRoute } from 'next'
|
|
import { DISALLOW } from '@/lib/publish'
|
|
|
|
// Static export: emits /robots.txt at build.
|
|
export const dynamic = 'force-static'
|
|
|
|
export default function robots(): MetadataRoute.Robots {
|
|
return {
|
|
// DISALLOW, not the route prefixes: lib/publish turns one prefix into the
|
|
// pair of RFC 9309 patterns that mean it, and this surface cannot spell a
|
|
// bare prefix. `Disallow: /auth` reads as a string and takes /authz with
|
|
// it; `/auth$` + `/auth/` is the same segment written in the crawler's own
|
|
// grammar.
|
|
//
|
|
// Private routes only. An unapproved route is deliberately fetchable, so
|
|
// that the `noindex` it carries is read — a Disallow here would be the one
|
|
// thing that prevents it.
|
|
rules: [{ userAgent: '*', allow: '/', disallow: [...DISALLOW] }],
|
|
sitemap: 'https://hanzo.ai/sitemap.xml',
|
|
host: 'https://hanzo.ai',
|
|
}
|
|
}
|