mirror of
https://github.com/hanzoai/next.js.git
synced 2026-09-24 17:00:51 +00:00
Luke SandbergandGitHub
2904599735
## Summary
Reworks how the turbo-tasks backend tracks modified tasks for persistence snapshots, reducing overhead and simplifying the snapshot lifecycle.
**Key changes:**
- **Replace `modified` DashMap with per-shard atomic counters + inline flags.** Instead of maintaining a separate `FxDashMap<TaskId, ModifiedState>` that mirrors every modification, track modifications via flags already on `TaskStorage` and use per-shard `AtomicU64` counters to skip unmodified shards during snapshot iteration. This eliminates a major source of memory overhead.
- The downside here is needing to scan the map. per shard counters enable early exits but we will still need to scan entire shards. For a large site that means scanning thousands of
- **Merge task cache writes into the snapshot pipeline.** New tasks now carry a `new_task` flag and their type hash in `SnapshotItem`, so task cache entries are written in the same batch as task data/meta — removing the separate `persisted_task_cache_log` (`Sharded<ChunkedVec<...>>`) and its associated locking.
- **Remove `local_is_partial` optimization.** The backing storage layer already short-circuits on empty databases, and new tasks eagerly set `restored` flags at allocation time, making this redundant.
- **Simplify `end_snapshot`.** Instead of a multi-pass retain/iterate/update cycle over the `modified` map, `end_snapshot` now just drains the small `snapshots` map (only tasks concurrently accessed during snapshot mode) and promotes their `modified_during_snapshot` flags.
- **Delete unused utilities.** Removes `Sharded`, `ChunkedVec` (from backing_storage), and `swap_retain` import now that they're no longer needed.
**Other cleanups:**
- `initialize_new_task` sets restored + new_task flags at allocation time for both persistent and transient tasks
- Fuzz test updated to use `active_tracking: true` and `StorageMode::ReadWrite`
- New KV storage test for batch write+flush+reopen pattern
- Minor fix: `SmallVec::into_boxed_slice()` instead of `into_vec().into_boxed_slice()`
## Build Benchmark Results
Measured over 9 runs (1 warm-up discarded), macOS, `TURBOPACK_PERSISTENT_CACHE=1`.
### Cold build (`rm -rf .next/`)
| | Time (avg) | Time (stddev) | MaxRSS (avg) |
|---|---|---|---|
| HEAD | 75.48s | 0.72s | 24,231 MiB |
| This PR | 75.20s | 1.98s | 23,829 MiB |
| **Delta** | −0.28s | — | **−402 MiB (−1.7%)** |
### Warm build (single file edit)
| | Time (avg) | MaxRSS (avg) |
|---|---|---|
| HEAD | 23.79s | 8,764 MiB |
| This PR | 23.67s | 8,766 MiB |
| **Delta** | −0.12s | flat |
Cold build time difference is within noise (< 1 stddev). The meaningful improvement is a **~400 MiB reduction in peak memory on cold builds**, consistent with the fixed overhead this PR targets. Warm builds are unaffected as expected.
53 lines
1.2 KiB
JavaScript
53 lines
1.2 KiB
JavaScript
const enableCaching = !!process.env.ENABLE_CACHING
|
|
|
|
/**
|
|
* @type {import('next').NextConfig}
|
|
*/
|
|
const nextConfig = {
|
|
turbopack: {
|
|
rules: {
|
|
'./app/**/page.{jsx,tsx}': {
|
|
loaders: ['./my-timestamp-loader.js'],
|
|
},
|
|
'./app/loader/page.tsx': {
|
|
loaders: ['./my-loader.js'],
|
|
},
|
|
'./pages/pages.tsx': {
|
|
loaders: ['./my-timestamp-loader.js'],
|
|
},
|
|
},
|
|
},
|
|
experimental: {
|
|
turbopackFileSystemCacheForBuild: enableCaching,
|
|
turbopackFileSystemCacheForDev: enableCaching,
|
|
},
|
|
env: {
|
|
NEXT_PUBLIC_CONFIG_ENV: 'hello world',
|
|
},
|
|
webpack(config, { dev }) {
|
|
config.module.rules.push({
|
|
test: /app(?:\/.*)?\/page\.[tj]sx|pages\/pages\.tsx/,
|
|
use: ['./my-timestamp-loader.js'],
|
|
})
|
|
config.module.rules.push({
|
|
test: /app\/loader(?:\/client)?\/page\.tsx/,
|
|
use: ['./my-loader.js'],
|
|
})
|
|
if (enableCaching) {
|
|
config.cache = Object.freeze({
|
|
type: 'memory',
|
|
})
|
|
}
|
|
if (dev) {
|
|
// Make webpack consider the build as large change which makes it filesystem cache it sooner
|
|
config.plugins.push((compiler) => {
|
|
compiler.__extra_delay = true
|
|
})
|
|
}
|
|
|
|
return config
|
|
},
|
|
}
|
|
|
|
module.exports = nextConfig
|