mirror of
https://github.com/hanzo-apps/hanzo.ai.git
synced 2026-10-30 14:58:08 +00:00
zeekay
4449d9234a
robots.txt states its Content Signals in both groups: search, ai-input and ai-train, which the groups already allow. /auth.md says how an agent gets a credential today (wallet sign-in, device flow, an API key) and that the auth.md registration protocol is not implemented. /.well-known/mcp/server-card.json describes the MCP endpoint at api.hanzo.ai/v1/mcp (SEP-2127, protocol 2026-07-28), and the API catalog names it. skill.mjs writes /skill.md again as an Agent Skill with its index (Agent Skills Discovery 0.2.0), the digest taken from the bytes it writes. wellknown.mjs copies subdirectories and refuses a server card that does not point at the API host.
59 lines
2.1 KiB
TypeScript
59 lines
2.1 KiB
TypeScript
import type { MetadataRoute } from 'next'
|
|
import { DISALLOW } from '@/lib/publish'
|
|
|
|
// Static export: emits /robots.txt at build.
|
|
export const dynamic = 'force-static'
|
|
|
|
/**
|
|
* The AI crawlers, named so that reading this file answers whether each one is
|
|
* welcome rather than leaving it to inference from `*`. A crawler that finds a
|
|
* group with its own name obeys that group alone (RFC 9309 §2.2.1), so each
|
|
* named group carries the same Disallow lines as `*`; written once, here.
|
|
*
|
|
* Bytespider is not named: whether it is welcome is the owner's decision, and
|
|
* until it is made it reads the `*` group like any other crawler.
|
|
*/
|
|
const AGENTS = [
|
|
'GPTBot',
|
|
'OAI-SearchBot',
|
|
'ChatGPT-User',
|
|
'ClaudeBot',
|
|
'Claude-User',
|
|
'Claude-SearchBot',
|
|
'anthropic-ai',
|
|
'PerplexityBot',
|
|
'Perplexity-User',
|
|
'Google-Extended',
|
|
'CCBot',
|
|
'Applebot-Extended',
|
|
]
|
|
|
|
/**
|
|
* What a crawler may do with what it reads (https://contentsignals.org). It
|
|
* states what the groups below already grant: search engines, the agents that
|
|
* fetch for a person (ChatGPT-User, Claude-User, Perplexity-User) and the
|
|
* training crawlers (GPTBot, Google-Extended, CCBot) are each allowed `/`. In
|
|
* both groups, because a named crawler reads its own group alone.
|
|
*/
|
|
const SIGNAL = { 'Content-Signal': 'search=yes, ai-input=yes, ai-train=yes' }
|
|
|
|
export default function robots(): MetadataRoute.Robots {
|
|
return {
|
|
// DISALLOW, not the route prefixes: lib/publish turns one prefix into the
|
|
// pair of RFC 9309 patterns that mean it, and this surface cannot spell a
|
|
// bare prefix. `Disallow: /auth` reads as a string and takes /authz with
|
|
// it; `/auth$` + `/auth/` is the same segment written in the crawler's own
|
|
// grammar.
|
|
//
|
|
// Private routes only. An unapproved route is deliberately fetchable, so
|
|
// that the `noindex` it carries is read — a Disallow here would be the one
|
|
// thing that prevents it.
|
|
rules: [
|
|
{ userAgent: '*', allow: '/', disallow: [...DISALLOW], other: SIGNAL },
|
|
{ userAgent: AGENTS, allow: '/', disallow: [...DISALLOW], other: SIGNAL },
|
|
],
|
|
sitemap: 'https://hanzo.ai/sitemap.xml',
|
|
host: 'https://hanzo.ai',
|
|
}
|
|
}
|