mirror of
https://github.com/hanzoai/next.js.git
synced 2026-10-30 14:46:38 +00:00
c2b4c0815c
This PRs unifies the caching story across the docs, making Cache Components the happy path, while still providing guidance to users in the old model. However, instead of explaining the old model and its caching layers, we've created a new guide focusing on what APIs to use and when. This follow-up PR aligns terminology across the docs: https://github.com/vercel/next.js/pull/90589 ## IA updates Getting Started section: - Improves Getting Started progression: - **Before:** CC → Fetching Data → Updating Data → Caching and Revalidating (old and new model mixed) - **After:** Fetching Data (Dynamic) → Mutating Data (Dynamic) → Caching with CC (Prerendering) → Revalidating with CC. - New: `caching.mdx` (CC-first) - Structure: - Enabling Cache Components - Data vs UI-level caching - Working with request time APIs - Passing request values to cached functions - Working with non-deterministic operations - Working with synchronous operations - How rendering works (PPR and static shell story) - New: `revalidating.mdx` (CC-first) - Explains how to use `cacheLife` and `cacheTag` Guides Section: - New: `caching-and-revalidating.mdx` (Previous Model) - For users who are not using CC, includes `fetch` options and route segment config - Moves route segment config options that don't apply to CC from API reference to this guide (for easy archiving in the future). - New: `migrating-to-cache-components.mdx` (WIP) - Del: `caching.mdx` 😌 ## Terminology We should remove caching layers from the docs. Users only needed to be exposed to them when they were configured independently, but the new CC APIs work across layers. To make it easier to review this PR, I'm consolidating terminology and fixing broken links in a new PR: https://github.com/vercel/next.js/pull/90589 --------- Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com> Co-authored-by: Joseph <[email protected]>
149 lines
3.1 KiB
Plaintext
149 lines
3.1 KiB
Plaintext
---
|
|
title: robots.txt
|
|
description: API Reference for robots.txt file.
|
|
---
|
|
|
|
Add or generate a `robots.txt` file that matches the [Robots Exclusion Standard](https://en.wikipedia.org/wiki/Robots.txt#Standard) in the **root** of `app` directory to tell search engine crawlers which URLs they can access on your site.
|
|
|
|
## Static `robots.txt`
|
|
|
|
```txt filename="app/robots.txt"
|
|
User-Agent: *
|
|
Allow: /
|
|
Disallow: /private/
|
|
|
|
Sitemap: https://acme.com/sitemap.xml
|
|
```
|
|
|
|
## Generate a Robots file
|
|
|
|
Add a `robots.js` or `robots.ts` file that returns a [`Robots` object](#robots-object).
|
|
|
|
> **Good to know**: `robots.js` is a special Route Handler that is cached by default unless it uses a [Request-time API](/docs/app/glossary#request-time-apis) or [dynamic config](/docs/app/guides/caching-without-cache-components#dynamic) option.
|
|
|
|
```ts filename="app/robots.ts" switcher
|
|
import type { MetadataRoute } from 'next'
|
|
|
|
export default function robots(): MetadataRoute.Robots {
|
|
return {
|
|
rules: {
|
|
userAgent: '*',
|
|
allow: '/',
|
|
disallow: '/private/',
|
|
},
|
|
sitemap: 'https://acme.com/sitemap.xml',
|
|
}
|
|
}
|
|
```
|
|
|
|
```js filename="app/robots.js" switcher
|
|
export default function robots() {
|
|
return {
|
|
rules: {
|
|
userAgent: '*',
|
|
allow: '/',
|
|
disallow: '/private/',
|
|
},
|
|
sitemap: 'https://acme.com/sitemap.xml',
|
|
}
|
|
}
|
|
```
|
|
|
|
Output:
|
|
|
|
```txt
|
|
User-Agent: *
|
|
Allow: /
|
|
Disallow: /private/
|
|
|
|
Sitemap: https://acme.com/sitemap.xml
|
|
```
|
|
|
|
### Customizing specific user agents
|
|
|
|
You can customize how individual search engine bots crawl your site by passing an array of user agents to the `rules` property. For example:
|
|
|
|
```ts filename="app/robots.ts" switcher
|
|
import type { MetadataRoute } from 'next'
|
|
|
|
export default function robots(): MetadataRoute.Robots {
|
|
return {
|
|
rules: [
|
|
{
|
|
userAgent: 'Googlebot',
|
|
allow: ['/'],
|
|
disallow: '/private/',
|
|
},
|
|
{
|
|
userAgent: ['Applebot', 'Bingbot'],
|
|
disallow: ['/'],
|
|
},
|
|
],
|
|
sitemap: 'https://acme.com/sitemap.xml',
|
|
}
|
|
}
|
|
```
|
|
|
|
```js filename="app/robots.js" switcher
|
|
export default function robots() {
|
|
return {
|
|
rules: [
|
|
{
|
|
userAgent: 'Googlebot',
|
|
allow: ['/'],
|
|
disallow: ['/private/'],
|
|
},
|
|
{
|
|
userAgent: ['Applebot', 'Bingbot'],
|
|
disallow: ['/'],
|
|
},
|
|
],
|
|
sitemap: 'https://acme.com/sitemap.xml',
|
|
}
|
|
}
|
|
```
|
|
|
|
Output:
|
|
|
|
```txt
|
|
User-Agent: Googlebot
|
|
Allow: /
|
|
Disallow: /private/
|
|
|
|
User-Agent: Applebot
|
|
Disallow: /
|
|
|
|
User-Agent: Bingbot
|
|
Disallow: /
|
|
|
|
Sitemap: https://acme.com/sitemap.xml
|
|
```
|
|
|
|
### Robots object
|
|
|
|
```tsx
|
|
type Robots = {
|
|
rules:
|
|
| {
|
|
userAgent?: string | string[]
|
|
allow?: string | string[]
|
|
disallow?: string | string[]
|
|
crawlDelay?: number
|
|
}
|
|
| Array<{
|
|
userAgent: string | string[]
|
|
allow?: string | string[]
|
|
disallow?: string | string[]
|
|
crawlDelay?: number
|
|
}>
|
|
sitemap?: string | string[]
|
|
host?: string
|
|
}
|
|
```
|
|
|
|
## Version History
|
|
|
|
| Version | Changes |
|
|
| --------- | -------------------- |
|
|
| `v13.3.0` | `robots` introduced. |
|