mirror of
https://github.com/hanzoai/next.js.git
synced 2026-09-24 17:00:51 +00:00
Luke SandbergandGitHub
2904599735
## Summary
Reworks how the turbo-tasks backend tracks modified tasks for persistence snapshots, reducing overhead and simplifying the snapshot lifecycle.
**Key changes:**
- **Replace `modified` DashMap with per-shard atomic counters + inline flags.** Instead of maintaining a separate `FxDashMap<TaskId, ModifiedState>` that mirrors every modification, track modifications via flags already on `TaskStorage` and use per-shard `AtomicU64` counters to skip unmodified shards during snapshot iteration. This eliminates a major source of memory overhead.
- The downside here is needing to scan the map. per shard counters enable early exits but we will still need to scan entire shards. For a large site that means scanning thousands of
- **Merge task cache writes into the snapshot pipeline.** New tasks now carry a `new_task` flag and their type hash in `SnapshotItem`, so task cache entries are written in the same batch as task data/meta — removing the separate `persisted_task_cache_log` (`Sharded<ChunkedVec<...>>`) and its associated locking.
- **Remove `local_is_partial` optimization.** The backing storage layer already short-circuits on empty databases, and new tasks eagerly set `restored` flags at allocation time, making this redundant.
- **Simplify `end_snapshot`.** Instead of a multi-pass retain/iterate/update cycle over the `modified` map, `end_snapshot` now just drains the small `snapshots` map (only tasks concurrently accessed during snapshot mode) and promotes their `modified_during_snapshot` flags.
- **Delete unused utilities.** Removes `Sharded`, `ChunkedVec` (from backing_storage), and `swap_retain` import now that they're no longer needed.
**Other cleanups:**
- `initialize_new_task` sets restored + new_task flags at allocation time for both persistent and transient tasks
- Fuzz test updated to use `active_tracking: true` and `StorageMode::ReadWrite`
- New KV storage test for batch write+flush+reopen pattern
- Minor fix: `SmallVec::into_boxed_slice()` instead of `into_vec().into_boxed_slice()`
## Build Benchmark Results
Measured over 9 runs (1 warm-up discarded), macOS, `TURBOPACK_PERSISTENT_CACHE=1`.
### Cold build (`rm -rf .next/`)
| | Time (avg) | Time (stddev) | MaxRSS (avg) |
|---|---|---|---|
| HEAD | 75.48s | 0.72s | 24,231 MiB |
| This PR | 75.20s | 1.98s | 23,829 MiB |
| **Delta** | −0.28s | — | **−402 MiB (−1.7%)** |
### Warm build (single file edit)
| | Time (avg) | MaxRSS (avg) |
|---|---|---|
| HEAD | 23.79s | 8,764 MiB |
| This PR | 23.67s | 8,766 MiB |
| **Delta** | −0.12s | flat |
Cold build time difference is within noise (< 1 stddev). The meaningful improvement is a **~400 MiB reduction in peak memory on cold builds**, consistent with the fixed overhead this PR targets. Warm builds are unaffected as expected.