100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Tobias KoppersandGitHub	b78f7d3f79	Turbopack: order CSS modules by chunk-group co-occurrence in linearize (#95579) ... ## Summary The graph-based CSS chunker's `linearize` step produces the global module order that `split_into_chunks` then cuts into chunks. It previously broke ties between ready modules using a weight-sorted stack. It now prefers the ready module that shares the most chunk groups with the previously placed module, so modules that are loaded together end up adjacent in the global order and split into better-aligned chunks. On a real world example this measurably lowers the modeled chunk-loading cost, with fewer requests and less over-fetched CSS. The shared-group count is read from edge weights (how many chunk groups two modules co-occur in). Since `make_acyclic` deletes edges to break cycles, it now returns the edges it cut so `linearize` can still count those conflicting-order co-occurrences — reconstructing the full co-occurrence without cloning the graph. ## Verification - `cargo test -p turbopack-core style_groups_graph::tests` (all pass) - Validated on a real world example that the resulting global order is deterministic and the modeled chunk-loading cost decreases. <!-- NEXT_JS_LLM_PR -->	2026-07-13 17:30:46 +00:00
Tobias KoppersandGitHub	1682685a35	Turbopack: replace cssChunking graph moduleFactorCost with weightDistribution (#95088) ... ## Summary Reworks the cost model of the experimental graph CSS chunking algorithm (`experimental.cssChunking: 'graph'`, Turbopack only) and retunes its defaults. - **Replaces the `moduleFactorCost` option with `weightDistribution`.** The old `moduleFactorCost` term penalized a chunk purely by `chunk_size / group_total_size`, which charged a chunk group even for CSS it fully needs and never actually measured overshipping. - **New per-group cost** is `chunk_group_weight * (chunk_size + request_cost)`, summed over the chunk groups that load a chunk, where `chunk_group_weight = group_total_size ^ (-weightDistribution)` is precomputed once per group. `weightDistribution = 0` weights every chunk group equally; higher values give smaller chunk groups a larger weight, so the algorithm overships less to small pages at the cost of more requests. The size weighting subsumes the explicit overship penalty, so the metric stays chunk-local and the greedy merger is unchanged. - **Retunes defaults:** `requestCost` `20000` → `100000` (bytes) and `weightDistribution` default `0.1`. Config shape (object form): `{ type: 'graph', requestCost?, weightDistribution? }`. The change is wired end to end through the config schema/types, `StyleGroupsAlgorithm::Graph`, and the chunking algorithm. ## Verification - `cargo test -p turbopack-core --lib style_groups_graph` (54 passed, incl. a new test asserting `weightDistribution` keeps an unneeded module out of a small group's chunk) - `pnpm test-start-turbo test/e2e/app-dir/css-order/css-order.test.ts` (243 passed, 12 todo) — confirms chunk shapes/request counts at the new defaults across the `graph` and `graph-object` modes <!-- NEXT_JS_LLM_PR -->	2026-06-29 17:53:24 +02:00
5 people	16db9d0544	Turbopack: add TURBOPACK_DEBUG_CSS_CHUNKING env var (#95080) ... ### What? Adds a `TURBOPACK_DEBUG_CSS_CHUNKING` environment variable. When set to a truthy value, the graph-based CSS chunker (`experimental.cssChunking: "graph"`) writes a JSON snapshot of its inputs and outputs to the current working directory on each invocation. ### Why? We are investigating problems with the new graph-based CSS chunking algorithm. The pipeline (`create_graph` → `make_acyclic` → `linearize` → `split_into_chunks`) is internal to `turbopack-core` and doesn't surface enough information through normal build output to debug bad chunking decisions post-hoc. A side-channel dump lets us reproduce and reason about a problematic build without instrumenting Rust on a user's machine. ### How? In `turbopack/crates/turbopack-core/src/module_graph/style_groups_graph/mod.rs`, between `split_into_chunks` and the result assembly: - Whether the dump is enabled is cached in a `static DEBUG_DUMP_ENABLED: LazyLock<bool>` so the env var is read exactly once per process. Truthy = anything other than unset, empty, `0`, or `false` (case-insensitive). - When enabled, every call to `compute_style_groups_graph` resolves `ident_string()` for every CSS module in parallel via `try_join` and writes a pretty-printed JSON file `turbopack-css-chunking-debug-<unix_ms>-<seq>.json` in the current working directory. The timestamp + atomic counter suffix ensures concurrent or repeated calls don't clobber each other. - Failures while writing the dump are logged to stderr and otherwise swallowed — a debug toggle must never fail a build. The JSON document contains: - `chunk_groups`: `string[][]` — module idents per chunk group, in the order the algorithm sees them. - `global_order`: `string[]` — the flat global order produced by `linearize`. - `global_order_chunks`: `string[][]` — the same modules grouped by the merged segments produced by `split_into_chunks` (e.g. `[["a","b"], ["c"], ["d","e","f"]]`). - `modules`: `[{ ident, size, style_type }]` — per-module metadata where `size` is the chunk item size in bytes (the same value used by the cost model) and `style_type` is `"GlobalStyle"` or `"IsolatedStyle"`. Verification: `cargo clippy -p turbopack-core --no-deps` is clean and the existing `style_groups_graph` test suite passes (53 tests). No user-facing config change, no docs change — the env var is intentionally undocumented and only meant for triage. Closes NEXT- Fixes # --------- Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-06-24 03:10:16 +02:00
5 people	97565d1592	DiskFileSystem: store Effect full_path as Arc<Path> (#94332) ... ### What? In the `DiskFileSystem` `write` and `write_link` implementations (`turbopack/crates/turbo-tasks-fs/src/lib.rs`), change the `full_path` field on the `WriteEffect` and `WriteLinkEffect` structs from `Arc<PathBuf>` to `Arc<Path>`. ### Why? `Arc<PathBuf>` stores two heap allocations and an extra indirection: the `Arc` allocation holds a `PathBuf` (`Vec<u8>`-like), which in turn points to the actual path bytes. `Arc<Path>` is a thin/wide-pointer to a single allocation containing the path bytes inline next to the `Arc` header, which is both smaller (no separate `PathBuf` capacity field) and removes one level of pointer chasing on every access. These effects are cloned per write side-effect and live until the effect is applied, so the saved allocation/indirection is a small but free win. ### How? - Change the `full_path` field type on `WriteEffect` and `WriteLinkEffect` from `Arc<PathBuf>` to `Arc<Path>`. - Update the two construction sites from `Arc::new(full_path)` to `Arc::from(full_path)`, which uses the standard library's `From<PathBuf> for Arc<Path>` impl to move the path bytes directly into a single `Arc<Path>` allocation. No call sites needed updating: all existing uses (`validate_path_length(&self.full_path)`, `self.inner.invalidate_from_write(&self.full_path)`, `self.full_path.as_os_str()`) already only need `&Path`, which is reached via `Deref` from both `Arc<PathBuf>` and `Arc<Path>`. Verified locally with `cargo check -p turbo-tasks-fs` and `cargo clippy -p turbo-tasks-fs --all-targets`. <!-- NEXT_JS_LLM_PR --> Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-06-03 17:24:04 +02:00
Tobias KoppersandGitHub	957520dbfb	DiskFileSystem: run write effects on a spawned task (#94140) ... ### What? Make `DiskFileSystem` write and symlink effects run on their own spawned tasks, so multiple pending writes can execute in parallel rather than serially on the caller's future. ### Why? Effects emitted by `DiskFileSystem::write` and `write_link` were previously applied inline within the caller's future. That meant the work for each effect ran sequentially in the awaiting task — writes did not overlap even though each one is largely I/O bound (path validation, lock acquisition, file write, fsync). ### How? - `WriteEffect` and `WriteLinkEffect` now derive `Clone` and own their data so they can be moved into a spawned task. `full_path` becomes `Arc<PathBuf>` to keep cloning cheap. - `apply_inner` takes `self` by value instead of `&self`. - `apply()` now calls `spawn(self.clone().apply_inner())` so the effect body runs on a dedicated turbo-tasks task. Multiple effects awaited concurrently will execute in parallel. Closes NEXT- Fixes # <!-- NEXT_JS_LLM_PR -->	2026-06-02 10:33:44 +02:00
5 people	5200800187	Include --port in next internal query-trace startup hint and help example (#93961) ... ### What? When the Turbopack trace server starts (via `next internal trace <file>`), it prints a hint suggesting users run `next internal query-trace --help` from another terminal. That suggestion doesn't include the actual MCP port the server is listening on, so anyone who copies it verbatim ends up querying the default port (`5748`) — which is wrong as soon as the server picked a non-default `--mcp-port`. This PR threads the running MCP port into both the startup hint and the `query-trace --help` output so the suggested command lines target the server that's actually running. Before: ``` Trace file opened Turbopack trace server started. View trace at https://trace.nextjs.org?port=5750 Query this trace from the command line: next internal query-trace --help Alternatively, connect an MCP client to http://127.0.0.1:5751/mcp ``` After: ``` Trace file opened Turbopack trace server started. View trace at https://trace.nextjs.org?port=5750 Query this trace from the command line: next internal query-trace --help --port 5751 Alternatively, connect an MCP client to http://127.0.0.1:5751/mcp ``` And `next internal query-trace --help --port 5751` now ends with: ``` Example: next internal query-trace --port 5751 --parent <id> ``` ### Why? The MCP HTTP port is dynamic (defaults to `wsPort + 1`, and `--mcp-port` can override it). Without echoing it into the hint, the suggested follow-up command silently connects to the wrong port and the user gets a confusing "Could not connect to trace server on port 5748" error instead of a useful response. Surfacing the right `--port` value at the point the user copies the command keeps the discovery flow self-consistent. ### How? - `packages/next/src/cli/internal/turbo-trace-server.ts`: the startup hint now interpolates the actual `httpPort` the MCP server bound to, so it prints `next internal query-trace --help --port <httpPort>`. - `packages/next/src/bin/next.ts`: the `query-trace` command gains an `.addHelpText('after', …)` block that reads the parsed `--port` option (falling back to the default `5748`) and appends an `Example:` section using that value. Passing `--port <port>` alongside `--help` therefore produces an immediately-runnable example for that specific server. The `--port` CLI option itself, its default, and parsing/validation are unchanged. ### Verification Built `next` and ran: - `next internal query-trace --help` → example shows `--port 5748` (default). - `next internal query-trace --help --port 5751` → example shows `--port 5751`. - `next internal query-trace --port 5751 --help` → example shows `--port 5751`. The existing `test/e2e/turbopack-trace-server-query/turbopack-trace-server.test.ts` suite still matches the substrings it asserts on (`next internal trace <file>` and `next internal query-trace --help`). <!-- NEXT_JS_LLM_PR --> Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-06-01 14:26:05 +02:00
5 people	11c865c1d6	Update ctor from 0.10 to 1.0.6 (#94045) ... ### What? Updates the `ctor` crate dependency from `0.10` to `1.0.6` in the workspace and migrates the two call sites to the new API. ### Why? We've been running into a rust-analyzer issue with `ctor` 0.10 that the newer release looks like it should fix. While bumping, the major version jump also brings in a number of stability and safety improvements (post-libc-init default priority, required `unsafe` marker, declarative re-export form, expanded platform support). ### How? `ctor` 1.x has several breaking changes that affect us: 1. **`#[ctor]` now requires an explicit `unsafe` marker.** Both call sites were updated: - `turbopack/crates/turbo-tasks-macros-tests/tests/trybuild.rs`: `#[ctor::ctor]` → `#[ctor::ctor(unsafe)]`. The function body already wraps `std::env::remove_var` in `unsafe`, so no further change was needed. - `turbopack/crates/turbo-tasks-macros/src/value_impl_macro.rs`: the proc-macro emits a vtable registration ctor for every `#[turbo_tasks::value_impl]`. Migrated to the new declarative form (see next bullet). 2. **`crate_path = ...` is deprecated in favor of `ctor::declarative::ctor!`.** The ctor 1.0.6 docs explicitly recommend the declarative macro for crates that re-export ctor (`turbo-tasks` re-exports it via `macro_helpers::ctor` so user crates don't need a direct dep). The emission in `value_impl_macro.rs` was rewritten from ```rust #[turbo_tasks::macro_helpers::ctor::ctor( crate_path = turbo_tasks::macro_helpers::ctor, )] #[allow(non_snake_case)] fn #vtable_register_ident() { ... } ``` to ```rust turbo_tasks::macro_helpers::ctor::declarative::ctor! { #[ctor(unsafe)] #[allow(non_snake_case)] fn #vtable_register_ident() { ... } } ``` 3. **Default priority changed.** In 0.x the default priority was `0` (run before C library init); in 1.x the default is `default` (≈500, after libc init). We don't specify a priority, and the registration ctors only push into a `Vec` guarded by a registry — they don't touch libc — so the later timing is safe and we accept the new default. The lazy `VTableRegistry::finalize` step (driven by the `VALUES` `LazyLock`) is unchanged, so visible ordering semantics for downstream code are preserved. 4. **Other 1.x breaking changes verified as non-issues for this repo:** - `dtor` was split into its own crate — not used here. - `priority = naked` syntax removed — not used. - MSRV bumps: base is 1.60, only the optional `priority` feature requires 1.85 — we don't enable `priority`. - `used_linker` / `no_warn_on_missing_unsafe` were replaced by `--cfg linktime_*` flags — neither was used here. #### Verification - `cargo check` / `cargo clippy` clean across `turbo-tasks`, `turbo-tasks-backend`, `turbo-tasks-macros`, `turbo-tasks-macros-tests`, and `turbopack-tests`. - `cargo test -p turbo-tasks-backend`: all tests pass (58 in the largest suite, 0 failures across all integration tests, 0 doc-test failures). - `cargo test -p turbopack-tests`: 218 + 87 tests pass, 0 failures, 1 ignored. #### Lockfile changes `Cargo.lock` reflects the new dep graph: `ctor` jumps to `1.0.6`, `link-section` to `0.17.2`, a new `linktime-proc-macro` is pulled in, and the old `ctor-proc-macro` / `dtor` / `dtor-proc-macro` entries are dropped (they were transitive deps of `ctor` 0.10). <!-- NEXT_JS_LLM_PR --> Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-06-01 14:25:48 +02:00
Tobias KoppersandGitHub	24e8c18a97	[turbopack] Store TaskDirtyCause in Dirtyness and pass to NativeFunction::span (#94057) ... ### What? Track the cause of task invalidation in turbo-tasks and surface it to the tracing span when a task executes, so traces can show *why* a task was re-run. A new `task_dirty_cause` Cargo feature gates the storage and parameter plumbing for this information. The existing `trace_task_dirty` feature now depends on it. ### Why? When investigating Turbopack performance and incremental-recompute behavior, traces show which tasks ran but not what invalidated them. Recording the `TaskDirtyCause` on the `turbo_tasks::function` span makes it possible to attribute task re-runs to their triggering change (cell update, output change, collectible change, etc.) directly from a trace. ### How? - Replace `Dirtyness::Dirty(TaskPriority)` with a struct variant `Dirty { parent_priority, cause: TaskDirtyCause }`. The `cause` field is only present when the `task_dirty_cause` feature is enabled. - Move `TaskDirtyCause` from `turbo-tasks-backend` into `turbo-tasks` so the span signature in `turbo-tasks` can reference it. - Replace `OutputChange { task_description }` with `OutputChange { function: FunctionId }`, and add a new `ResolveOutputChange { function: FunctionId }` variant used when the task's output is `OutputValue::Output(_)`. This avoids constructing a `String` for the cause at invalidation time. - `NativeFunction::span` now accepts a feature-gated `cause: Option<&TaskDirtyCause>` argument and records it as a `cause` field on the `turbo_tasks::function` span. The cause is read from the dirty state in `try_start_task_execution` before execution starts. - Split the `trace_task_dirty` feature: the new `task_dirty_cause` feature owns the storage/parameter plumbing, and `trace_task_dirty` now depends on it. This lets consumers opt into the storage cost without the full tracing overhead, and keeps the default build unaffected. Closes PACK- <!-- NEXT_JS_LLM_PR -->	2026-05-29 07:59:35 +00:00
5 people	859c4edcf0	Produce valid file URLs for import.meta.url on Windows in Turbopack (#94179) ... ### What? `import.meta.url` evaluated in a Turbopack-compiled module on Windows produced an invalid file URI: the path portion contained backslashes (`\`) and was not URL-encoded, e.g. ``` file://C:\Users\me\project\apps\web\app\page.tsx ``` This affects any code that hands the value to `URL`, `fileURLToPath`, source maps, dev tools, or anything else that expects a [valid file URI](https://en.wikipedia.org/wiki/File_URI_scheme). This PR makes `import.meta.url` return a correct, percent-encoded file URI on all platforms, e.g. `file:///C:/Users/me/project/apps/web/app/page.tsx`. ### Why? The existing codegen in `references/esm/meta.rs` emitted ```js `file://${__turbopack_context__.P("<rel/path>")}` ``` where `__turbopack_context__.P` is the runtime helper `resolveAbsolutePath`. On the Node-side runtime (`shared-node/node-externals-utils.ts`) that helper returns `path.join(ABSOLUTE_ROOT, modulePath)`, which is OS-native — so on Windows it returns a backslash-separated path with no percent-encoding. The result was then concatenated into a `file://...` template literal, producing an invalid URI. The initial hint that `turbopack-core/src/source_map/utils.rs` was the culprit turned out to be a false lead: that code path uses `FileSystemPath::try_join`, which already normalizes `\` to `/` on Windows. The bug is in the codegen + runtime helper for `import.meta.url`, not in source-map handling. ### How? - Introduce a new runtime function `__turbopack_resolve_file_url__` (shortcut `F`) whose contract is to return a complete `file://` URI for a given relative module path. - Node implementation (`shared-node/node-externals-utils.ts`): uses `url.pathToFileURL(resolveAbsolutePath(modulePath)).href`. `pathToFileURL` handles drive letters (`file:///C:/...`), normalizes slashes, and percent-encodes path segments. - Browser implementation (`browser/runtime/base/runtime-base.ts`): returns `` `file:///ROOT/${modulePath ?? ''}` `` — the browser runtime intentionally does not expose the real filesystem path, so it returns the same stable placeholder as before, just with a valid `file:///` prefix instead of having callers concatenate one. - Change the `import.meta.url` codegen in `references/esm/meta.rs` to call `__turbopack_resolve_file_url__($formatted)` directly instead of concatenating a `file://` prefix to the absolute path. The pre-existing `encode_path` is kept only to keep the embedded JS string literal safe (the runtime helper is responsible for the URI-level encoding). - Update `TurbopackBaseContext` and add a `ResolveFileUrl` type alias in `runtime-types.d.ts` so the new shortcut is typed. - Register the new shortcut in `TURBOPACK_RUNTIME_FUNCTION_SHORTCUTS` so user code can also reference `__turbopack_resolve_file_url__` (mirroring the existing `__turbopack_resolve_absolute_path__`). `P` is left in place and unchanged because it's still used elsewhere. - Regenerate the affected Turbopack snapshot fixtures (`import-meta/*`, `comptime/typeof`, `runtime/default_*_runtime`, `workers/*`, `debug-ids/*`, etc.). The functional change in the generated code is uniform: `` `file://${__turbopack_context__.P(<rel>)}` `` → `__turbopack_context__.F(<rel>)`. - Drop the `// TODO: These file URIs are wrong on turbopack+windows` comment and the `.replaceAll('\\', '/')` workarounds from `test/e2e/app-dir/non-root-project-monorepo/non-root-project-monorepo.test.ts`, so the assertion now verifies that the URI is correctly slashed and encoded on all platforms. ### Verification - `cargo clippy --workspace --all-targets` — clean - `cargo fmt --check` — clean - `cargo test -p turbopack-tests --test snapshot` — 87/87 pass - `cargo test -p turbopack-tests --test execution` — 219/219 pass (1 ignored) - `pnpm test-dev-turbo test/e2e/app-dir/non-root-project-monorepo/non-root-project-monorepo.test.ts` — 9/9 pass - `pnpm test-start-turbo test/e2e/app-dir/non-root-project-monorepo/non-root-project-monorepo.test.ts` — 6/6 pass - `pnpm test-dev-webpack test/e2e/app-dir/non-root-project-monorepo/non-root-project-monorepo.test.ts` — 9/9 pass (verifying webpack path still works) Windows itself can't be exercised in the sandbox, but the new code path no longer depends on `path.sep` or any OS-native string manipulation for the URI: `url.pathToFileURL` is the canonical Node API for producing a file URI from any OS-native path, and the test assertion has been tightened so that a regression on Windows would fail in CI rather than be papered over. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-29 07:34:41 +00:00
Tobias KoppersandGitHub	fda8a605dd	ServerActionsGraph: pass ModuleGraphLayer as OperationVc (#94141) ... ### What? Threads the layer module graph through `ServerActionsGraph` and `map_server_actions` as an `OperationVc` instead of a resolved `Vc`, calling `connect()` at each consumer. ### Why? Fixes the invalidation priority of `map_server_actions` so it is executed after the module graph on incremental builds. Previously, because the graph was resolved at construction time, the dependency ordering caused `map_server_actions` to be re-run before the module graph on incremental rebuilds. ### How? By passing the layer graph as an `OperationVc` and calling `connect()` within each consumer task, the graph is resolved within the consuming task rather than at construction time. This restores the intended invalidation/scheduling order between the module graph and `map_server_actions`. <!-- NEXT_JS_LLM_PR -->	2026-05-27 03:57:02 +02:00
Tobias KoppersandGitHub	1b77dba691	Turbopack: Fix unsound IntoIterator for ReadRef<T> (#94122) ... ### What? Replaces the unsound by-value `IntoIterator` impl for `ReadRef<T>` in `turbo-tasks` with a sound clone-free variant, and adapts the callers across `turbopack-*` and `next-api`. ### Why? The previous impl used `transmute_copy` to fabricate `&'static`-typed items so it could expose them through the standard `Iterator` trait. Those references were only really valid as long as the `ReadRef` inside the iterator stayed alive — but `Iterator::Item` is a fixed associated type, so once the items were stashed in futures, `Vec`s, or `serde_json` map keys, the lifetime was completely unenforced. This produced a latent use-after-free whenever something else (turbo-tasks cell eviction, an intermediate `Drop`, etc.) released the underlying storage between the iteration site and the next dereference. The observed symptom was a panic in `RcStr::as_str` during JSON serialization of `AssetHashesManifestAsset`'s manifest: ``` thread 'tokio-rt-worker' panicked at turbopack/crates/turbo-rcstr/src/lib.rs:132:52: range end index 13 out of range for slice of length 7 ``` The byte read at the inline-length position was junk left over from freed/reused memory — `len = 13` is unreachable for any legitimately-constructed inline `RcStr` (max inline length is 7 on 64-bit). The bug site was `crates/next-api/src/project_asset_hashes_manifest.rs`, which consumed an `OutputAssetsWithPaths` `ReadRef`, kept `&RcStr` references in `asset_paths` past the `try_join` that dropped the iterator, then serialized them. ### How? **`turbopack/crates/turbo-tasks/src/read_ref.rs`** — new by-value impl: ```rust pub struct ReadRefIter<T, I, J> where T: VcValueType, I: Copy + 'static, J: Iterator<Item = &'static I>, { iter: J, _read_ref: ReadRef<T>, } impl<T, I, J> Iterator for ReadRefIter<T, I, J> /* … */ { type Item = I; fn next(&mut self) -> Option<I> { self.iter.next().copied() } } impl<T, I, J> IntoIterator for ReadRef<T> where T: VcValueType, I: Copy + 'static, J: Iterator<Item = &'static I> + 'static, &'static VcReadTarget<T>: IntoIterator<Item = &'static I, IntoIter = J>, { type Item = I; type IntoIter = ReadRefIter<T, I, J>; fn into_iter(self) -> Self::IntoIter { let r: &VcReadTarget<T> = &self; // SAFETY: the fabricated `&'static` reference is only stored inside // `iter`, which lives inside the returned `ReadRefIter` alongside // the `ReadRef` that owns the data. `next()` only ever yields // `Copy`-ed-out values — no reference (with the fake `'static` // lifetime or otherwise) ever leaves the iterator. Struct-field drop // order (`iter` then `_read_ref`) drops the borrow before the // backing storage. let r = unsafe { std::mem::transmute::<&VcReadTarget<T>, &'static VcReadTarget<T>>(r) }; ReadRefIter { iter: r.into_iter(), _read_ref: self } } } ``` Key properties: - **No cloning.** Setup is one borrow + `transmute`; `next()` is `Option::copied()` (bitwise copy via the `Copy` bound), not `Clone::clone`. Nothing in the iterator clones the backing collection or its elements. - **Contained `unsafe`.** The fake `'static` reference never leaves `ReadRefIter`. `Iterator::next` yields `I` by value, so the lifetime never escapes into futures, `Vec`s, or other persistence outside the iterator. - **Drop order safe.** Struct fields drop in declaration order: `iter` (and any borrows it holds) drops before `_read_ref` (the backing `Arc`). - **`Copy` bound.** The impl is restricted to element types that are `Copy` — `ResolvedVc<_>`, integer ids, owned-tuple-of-`Copy`, etc. For non-`Copy` element types (`RcStr`, `FileSystemPath`, `PatternMatch`, `(String, _)`, `(ModuleId, ReadRef<_>)`, …) callers iterate by reference via the existing `IntoIterator for &'a ReadRef<T>` impl (`for x in &read_ref` or `read_ref.iter()`). The original buggy site in `project_asset_hashes_manifest.rs` now uses `output_assets.iter()` and keeps `&'a RcStr` references in the manifest struct. The borrow checker now enforces the lifetime that used to be faked via `transmute` — `output_assets` outlives the references because nothing consumes it, and there are no clones at the call site either. **Caller adjustments.** Touching the impl forced a sweep of all call sites that were implicitly leaning on the unsound shape (yielding `&'static`-typed items as a stand-in for owned items). The fixes fall into a small number of categories: - Drop redundant `.copied()` / `.cloned()` / `|&x| f(x)` patterns after `into_iter()` (items are owned `Copy` values now, no need to deref-and-copy). - Switch non-`Copy` element iteration to `&read_ref` / `read_ref.iter()` (e.g. `PatternMatches`, `CodeAndIds`, `UnresolvedUrlReferences`, `GraphEntries`, `Vec<RcStr>`). - Reshape `crates/next-api/src/paths.rs` helpers from `impl IntoIterator<Item = &ResolvedVc<_>>` to `impl IntoIterator<Item = ResolvedVc<_>>` — `ResolvedVc` is `Copy`, so by-value is the natural shape and it composes directly with the new by-value `ReadRef::into_iter`. Callers in `app.rs`, `pages.rs`, `middleware.rs`, `instrumentation.rs`, `font.rs` updated to match (either passing the `ReadRef`/`Vec` directly, or `.iter().copied()` for borrowed sources). - A few small follow-ups: `for (key, EndpointGroup { primary, .. }) in &entrypoint_groups` in `routes_hashes_manifest.rs` (with a borrowed `&'l str` key in the manifest); `compute_async_module_info_single(graph, result)` (no `*graph`, it's already `Copy`); `&(ty, batch)` → `(ty, batch)` destructures in `chunking/mod.rs`. ### Testing - `cac` clean across the workspace. - `ca clippy --all-targets` clean. - `ca test -p turbo-tasks-backend` — all unit + integration tests pass. - `ca test -p turbopack-tests --tests` — execution snapshot suite (218 passed, 0 failed, 1 ignored) and snapshot suite (87 passed, 0 failed). Closes NEXT- Fixes #	2026-05-26 08:40:30 -07:00
Tobias KoppersandGitHub	0070514701	Turbopack: make available_modules an OperationVc and simplify merged module bookkeeping (#94055) ... ### What? Reworks how `chunk_group_content` exposes `available_modules` for downstream chunking, and consolidates the bookkeeping for merged modules in `ChunkGroup`. - Splits `ChunkGroupContent` into a `turbo_tasks::value` `ChunkGroupContentInner` (carrying `chunkable_items`, `batch_groups`, `async_modules`, `traced_modules`, and `available_modules`) plus a thin wrapper that pairs it with the resulting `AvailabilityInfo`. `chunk_group_content_operation` now runs entirely inside `turbo_tasks`, allowing `AvailableModules` to store its set as an `OperationVc<AvailableModulesSet>` and `.connect()` it on read. - Replaces the separate `included` set and `should_create_chunk_item_for` method on `MergedModuleInfo` with an `Option<ChunkableModule>` value type on the `replacements` map. Call sites now decide all three cases (replace / skip / keep) in one match via `should_replace_module`. ### Why? - `AvailableModules::snapshot` did not correctly derive its invalidation priority, which caused double invalidations: the snapshot was treated as a low-priority dependency even though it gated chunking work, so when its inputs changed turbo-tasks would invalidate it after dependent computations had already started, triggering them a second time. Exposing `available_modules` as an `OperationVc` makes `AvailableModules` inherit the correct invalidation priority from the operation that produces it, which removes the double-invalidation. - The previous `MergedModuleInfo` API had two parallel data structures (`replacements` and `included`) and a helper that combined them. Folding `included` into the value of `replacements` removes the duplication, makes the three states (replace, skip-because-merged, keep) explicit, and shrinks the call-site logic in `chunk_group.rs`. ### How? **`available_modules` as an `OperationVc`:** - New `ChunkGroupContentInner` is a `turbo_tasks::value` holding the chunkable items, batch groups, async/traced modules, and an `OperationVc<AvailableModulesSet>` for `available_modules`. - `ChunkGroupContent` becomes a plain struct wrapping `ResolvedVc<ChunkGroupContentInner>` together with the `AvailabilityInfo` computed for the group. - `chunk_group_content_operation` is an `OperationVc`-producing function so its inner `available_modules` can be returned as an `OperationVc<AvailableModulesSet>`. `AvailableModules` now stores that operation Vc and `.connect()`s it whenever the set is read, so consumers reach the set through the operation and inherit its priority. - Browser and Node.js chunking contexts are updated to thread the new shape through `make_chunk_group` and friends. **`MergedModuleInfo` simplification:** - `replacements` changes from `HashMap<ChunkableModule, ChunkableModule>` to `HashMap<ChunkableModule, Option<ChunkableModule>>`: `Some(m)` means "replace with `m`", `None` means "skip — already included in another merged module", absent means "keep as-is". - The old `included` set and `should_create_chunk_item_for` helper are removed; `should_replace_module` returns the three-way outcome and consumers match on it once. - Call sites in `chunk_group.rs` are restructured around the single match so the logic is no longer split across two lookups. ### Notes - No user-facing behavior change is intended — this is an internal refactor in `turbopack-core` (`chunk_group_content`, `available_modules`, `chunk_group`) and the browser/nodejs chunking contexts. - Existing chunking tests in `turbopack-core` exercise the new shape. <!-- NEXT_JS_LLM_PR -->	2026-05-26 13:10:04 +02:00
5 people	a80e0d5668	Fix errors.json merge driver to handle gaps in error codes (#93829) ... ### What? Fixes the `errors.json` git merge driver (`scripts/merge-errors-json/merge.mjs`) so that it allocates new error ids based on the largest existing numeric key rather than the entry count. When the current file has gaps in its numeric keys (e.g. retired error codes), the previous logic produced ids that collided with existing entries and silently overwrote them during a rebase/merge. ### Why? The driver computed the next id with: ```js let nextKey = Object.keys(result).length + 1 ``` This only works when ids are dense (`1..N` with no holes). `packages/next/errors.json` has accumulated gaps over time — for example missing ids around 1232–1234 and 1246 on `canary`. When a branch was rebased onto `canary` and the driver replayed a commit that added new entries, the computed `nextKey` (`length + 1`) pointed at an id that was already in use higher up in the map. The replayed entries then clobbered existing canary messages (e.g. "Route encountered %s during the initial render" and "in a Client Component" at 1244/1245), corrupting `errors.json` without raising a conflict. Because the driver writes the result directly and exits 0, this kind of corruption is invisible at rebase time and only surfaces later when the displaced messages start producing wrong/duplicate codes at runtime. ### How? - Introduce a `getMaxNumericKey(map)` helper that scans the keys of the merged map, parses each as an integer, and returns the largest value (or `0` if there are no numeric keys). - Use `getMaxNumericKey(result) + 1` as the starting `nextKey` in `mergeErrors`, so newly added messages are always appended above the highest currently-used id, regardless of gaps. Behavior for the dense / no-gap case is unchanged: `max + 1 === length + 1` when ids are `1..N` contiguous, so all three existing scenarios in `test/scripts/merge-errors-json/merge-errors-json.test.ts` (basic conflict, empty base, stacked PRs) continue to pass. No changes to `errors.json` itself are included here; this PR only fixes the driver going forward. <!-- NEXT_JS_LLM_PR --> Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-25 20:29:01 +02:00
5 people	148c7ff020	Rename TaskExecutionReason::Initial to Root (#93922) ... ### What? Renames the `TaskExecutionReason::Initial` variant to `TaskExecutionReason::Root` in turbo-tasks, and updates its `as_str()` representation from `"initial"` to `"root"`. ### Why? The variant's doc comment already describes it as "A root task was initially scheduled and is executed". The name `Initial` is misleading because it suggests "first execution of any task", which overlaps semantically with other variants like `ActivateInitial` (a task activated for the first time with no output yet). `Root` accurately reflects that this reason is specifically used when a root task is scheduled, making the enum easier to read and reason about at call sites and in traces. ### How? - `turbopack/crates/turbo-tasks/src/task_execution_reason.rs`: rename the enum variant `Initial` → `Root` and update the `as_str()` mapping to return `"root"`. - `turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs`: update the single construction site that used `TaskExecutionReason::Initial` when scheduling a root task. No behavioral changes — this is a pure rename. The other `*Initial` variant (`ActivateInitial`) is intentionally left untouched because it describes a different concept (first-time activation of a non-root task). Verified with `cargo check` and `cargo clippy` on `turbo-tasks` and `turbo-tasks-backend`. Closes NEXT- Fixes # Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-18 13:43:54 -07:00
3 people	75d2ed53f7	turbopack: emit assets before duplicate check in emit_assets (#93875) ... ### What? Reworks `emit_assets` in `crates/next-core/src/emit.rs` so that: - The actual `emit()` / `emit_rebase()` write runs **before** `check_duplicates()`. - Per-duplicate diff checks inside `check_duplicates` run in parallel. - At most one `EmitConflictIssue` is emitted per asset path, even when multiple duplicates conflict. - The conflict-issue emission is wrapped in a `turbo_tasks::function` and reported via `as_side_effect()`. ### Why? Under eventual consistency, all children might be removed from `emit_assets`. This causes all `content()` and `emit()` tasks being inactive and keeping the stale value (which is potentially an error). On recomputation of `emit_assets()` this only makes very few `content()` tasks active, as tasks are still in a stale error state and make `emit_assets()` exit early. So it takes N recomputations of `emit_assets()` to be up-to-date, where N is the number of duplicate assets. This can slow down builds drastically and did cause build time up to 30min. Emitting the asset first ensures the file is written to disk even if the subsequent duplicate diagnostic crashes under eventual consistency. Doing the diff checks in parallel and folding conflicts into a single issue keeps the diagnostic path cheap and avoids noisy duplicate output for the same path. Tracking the issue emission as a turbo-tasks side effect means it is properly attributed to a task instead of being emitted ad-hoc from `emit_assets`. ### How? - Swap the order in both branches of `emit_assets` (server-side `emit` and client-side `emit_rebase`) so the write happens first, then `check_duplicates` runs as a diagnostic afterwards. A comment is left explaining the ordering requirement. - `check_duplicates` now returns `Result<()>` instead of returning the picked asset. The caller picks `assets.first()` directly, since `check_duplicates` only validated equivalence — it didn't choose a different asset. - The inner loop in `check_duplicates` uses `iter.map(async |next| { ... }).try_flat_join().await?` to run the diff comparisons in parallel and collect conflict details. `ext` is computed once before the loop and cloned into each task. - Only the first conflict from the collected list is reported, instead of one issue per conflicting asset. - The `EmitConflictIssue` emission is moved into a local `#[turbo_tasks::function] fn emit_conflict_issue(...)` invoked via `.as_side_effect().await?`, so the issue is tracked as a task side effect. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-05-18 09:02:30 +00:00
5 people	e2fca6c66a	[Turbopack] Add graph-based CSS chunking algorithm behind experimental.cssChunking: "graph" (#93606) ... ### What? Adds an alternative CSS chunking algorithm to Turbopack, opted into via: ```js // next.config.js module.exports = { experimental: { cssChunking: 'graph', // or, with explicit cost overrides: // cssChunking: { type: 'graph', requestCost: 20_000, moduleFactorCost: 1 }, }, } ``` The new algorithm is **off by default** — Turbopack still uses the existing "loose"/dependencies algorithm unless this flag is set, so this PR is a pure addition for users that don't opt in. While we were here, the `experimental.cssChunking` shape was also generalized so every existing string accepts an object form too: | Value | Bundler | Notes | |---|---|---| | `true` / `'loose'` / `{ type: 'loose' }` | both | default heuristic-based chunking | | `'strict'` / `{ type: 'strict' }` | webpack | unchanged | | `false` | webpack | unchanged (one chunk per CSS module) | | `'graph'` / `{ type: 'graph', requestCost?, moduleFactorCost? }` | Turbopack | new | Cross-bundler combinations are rejected at config-validation time: - `'graph'` with webpack throws. - `'strict'` and `false` with Turbopack throw. ### Why? The existing Turbopack CSS chunker (loose / dependencies) is good at preserving CSS ordering but doesn't share chunks across pages well — every page tends to load its own chunk per CSS module, which scales poorly for apps with many pages and shared component libraries. The new "graph" algorithm models the per-chunk-group CSS ordering as a weighted DAG over modules, then greedily merges adjacent runs in the global topological order whenever the merge reduces total cost. The cost model charges every CSS request and overshipped byte, with two tunable knobs (`requestCost` and `moduleFactorCost`). **Trade-off vs. the loose default.** With the default cost parameters (`requestCost: 20_000`, `moduleFactorCost: 1`) the graph algorithm typically ships **less CSS per chunk group at the cost of more requests** than the loose algorithm. The cost model is tuned to avoid overshipping unrelated CSS into pages that don't need it; on apps where the loose algorithm was already collapsing a lot into one big chunk that some pages didn't actually use, the graph algorithm will split it. Apps that prefer fewer requests can raise `requestCost`; apps that prefer less overshipping can raise `moduleFactorCost`. This is opt-in and Turbopack-only because: - The cost model is sensitive to per-app properties (number of pages, size distribution of CSS modules, …) — keeping it experimental gives us room to tune defaults from real usage. - Webpack already has its own `CssChunkingPlugin` and `'strict'` mode that cover the equivalent design space; we don't want to fork that. ### Performance Measured on `vercel.com` (the full graph algorithm spans `create_graph → make_acyclic → linearize → split_into_chunks → assemble`): - **~3s** end-to-end for the synchronous chunking pipeline on a realistic production input. Implementation choices that matter for that throughput: - Tarjan SCC uses `Vec<u32>` / `Vec<bool>` scratch arrays indexed by `NodeIndex` — no hashing on `indices` / `lowlinks` / `on_stack`. - `make_acyclic` batches multiple cuts per SCC pass by seeding successive short-cycle searches at the previous cut's target, only re-running Tarjan when no further cycle is reachable from the seed. - `find_short_cycle` is a bidirectional Dijkstra over a `BinaryHeap` with predecessor pointers (no path cloning) and skips its refinement loop for trivial 2-cycles. - `split_into_chunks` picks the next merge from a `BinaryHeap` keyed on the cost delta instead of an O(N) linear scan per merge. - `chunk_cost` reads a once-built `module_to_groups` inverse index instead of scanning every chunk group on every call; the GlobalStyle leakage check uses binary search on the inverse index rather than scanning each group's module list. ### How? #### Module layout (`turbopack/crates/turbopack-core/src/module_graph/`) The two algorithms are deliberately split so neither imports from the other: - `style_groups/` — algorithm-neutral output types (`StyleGroups`, `StyleItemInfo`, `make_style_groups`). Both algorithms produce these. - `style_groups_loose/` — the existing ("loose") algorithm plus the shared config types (`StyleGroupsAlgorithm`, `StyleGroupsConfig`, `F32TaskInput`). - `style_groups_graph/` — the new algorithm. Pure Rust, no `Vc`, with `petgraph::DiGraph` plus a thin `SubgraphView` wrapper and a small `ReadonlyGraph` trait that lets the same pipeline run against either a `&DiGraph` or a filtered view of one SCC. #### Algorithm ```text create_graph → make_acyclic → linearize → split_into_chunks → assemble batches ``` 1. **`create_graph`** — for each chunk group, every `(later, earlier)` pair inside the group's CSS-module list becomes an edge `later → earlier` (weight 1, accumulated). Heavy edges = strong co-occurrence. 2. **`make_acyclic`** — co-occurrence almost always introduces cycles; each multi-node SCC has its lowest-weight cycle edge cut until the graph is a DAG. 3. **`linearize`** — Kahn-style topological sort with a tie-break on edge weight, so strongly co-occurring modules end up adjacent in the global order. 4. **`split_into_chunks`** — greedy bottom-up merger over the global order. At every active split point we score the merge as `cost(merged) - cost(left) - cost(right)`, take the most-negative score from a min-heap, and repeat until no merge would reduce cost. `max_chunk_size` and "global CSS must not leak into unrelated chunk groups" are enforced as `+infinity` cost. The cost model is: ```text cost_per_group(chunk, group) = chunk_size + (chunk_size / group_total_size) * module_factor_cost + request_cost ``` summed over the chunk groups that load the chunk. #### Wiring - `StyleGroups::shared_chunk_items` is a `FxIndexMap<ChunkItemWithAsyncModuleInfo, StyleItemInfo>` where `StyleItemInfo { order: Option<u32>, batch: Option<…> }`. The graph algorithm fills `order` so `style_production.rs` can stable-sort chunks globally; the legacy algorithm leaves `order = None`, which makes the sort a no-op for it. `flatten_and_sort` returns the `StyleItemInfo` references alongside each chunk item so the per-item loop doesn't re-query the map. - A new `StyleGroupsAlgorithm` enum on `ChunkingConfig` selects the algorithm at chunking time; `ModuleGraph::style_groups` dispatches to either `compute_style_groups` (existing) or `compute_style_groups_graph` (new). - `next-core` exposes `NextConfig::css_chunking() -> Vc<CssChunkingAlgorithm>` resolving the JS `experimental.cssChunking` to the Rust enum, with cost defaults applied (`requestCost: 20_000`, `moduleFactorCost: 1`). All three chunking-context constructors (`next_client`, `next_edge`, `next_server`) thread it through. #### Configuration - `experimental.cssChunking` zod schema accepts the new shapes; cost params are `z.number().nonnegative().finite().optional()`. - `config-shared.ts` exports a `CssChunkingConfig` type alias and a `resolveCssChunkingMode(value)` helper that normalizes any input to one of `'off' | 'loose' | 'strict' | 'graph'`. Both `webpack-config.ts` (plugin wiring) and `config.ts` (bundler-compat validation) use the helper. - New `errors.json` entries for the three bundler-compatibility validation errors (E1193 graph-on-webpack, E1194 strict-on-Turbopack, E1195 false-on-Turbopack). #### Tests - 53 Rust unit tests in `style_groups_graph/tests.rs` cover `create_graph`, Tarjan SCC, `find_short_cycle` (bidirectional Dijkstra), `make_acyclic`, `linearize`, `split_into_chunks`, and end-to-end pipeline scenarios. - `test/e2e/app-dir/css-order/css-order.test.ts` is parametrised over `[label, value]` pairs. The Turbopack matrix now includes `'graph'` and an object-form `{ type: 'graph', requestCost: 1, moduleFactorCost: 1 }` in addition to the existing default. Per-page expectations grew a `requests` object encoding distinct request counts for `loose` and `graph` where they differ. - A new `sandwich` e2e fixture (`/sandwich/a`, `/sandwich/b`) exercises the case where two pages share a leading and trailing chunk around a unique middle stylesheet — including a global stylesheet that the algorithm must not leak into unrelated chunk groups. The graph algorithm hits the optimal 3 chunks per page on this fixture; loose mode falls short. #### Documentation - `ExperimentalConfig.cssChunking` JSDoc describes every accepted shape and what each cost knob does. - The `style_groups_graph` module-level docs describe the pipeline, cost model and constraints with diagrams. Closes NEXT- <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-18 08:45:03 +00:00
Tobias KoppersandGitHub	703d4da43e	turbo-tasks: prioritize recomputed tasks with Recomputation priority (#93876) ... ### What? Introduces a new `Recomputation` priority in `TaskPriority` that schedules recomputed tasks (tasks re-run because their output or cell wasn't available when read) ahead of all invalidation phases. Also documents every `TaskExecutionReason` variant and removes the unused `Local` variant. ### Why? When a task is recomputed because a downstream consumer needed its output/cell, that recomputation should run before lower-priority invalidations so the consumer can make progress. Previously these recomputations were scheduled with `Initial` priority, which placed them alongside fresh work and behind ongoing invalidation phases, hurting latency for the task actually being awaited. Care is needed so the new priority doesn't spread virally: any invalidation triggered during a `Recomputation` task must not itself be tagged `Recomputation`, otherwise the priority would leak across the graph and break ordering. The change therefore resets the priority to `Initial` for invalidations triggered from a recomputation context. ### How? - `turbo-tasks/src/manager.rs`: Add `TaskPriority::Recomputation` variant, ordered before `Invalidation`. Update `Display` and `update` impls. - `turbo-tasks-backend/src/backend/mod.rs`: Schedule output-not-available and cell-not-available recomputations with `TaskPriority::Recomputation` instead of `Initial`. - `turbo-tasks-backend/src/backend/operation/invalidate.rs`: When `make_task_dirty_internal` runs under a parent task with `Recomputation` priority, downgrade the propagated priority to `Initial` so the recomputation marker does not spread. Also tighten the "already dirty" branch so a stored `Initial` priority is always replaced by a more specific parent priority. - `turbo-tasks/src/task_execution_reason.rs`: Document each variant and drop the unused `Local` variant (and its `as_str` arm). Closes NEXT- <!-- NEXT_JS_LLM_PR -->	2026-05-17 19:36:07 +00:00
5 people	eab3ab87da	Add next internal static-routes-info CLI command (#93399) ... ### What? Adds a new internal CLI subcommand: ``` next internal static-routes-info [directory] [options] ``` It runs against an already-built Next.js app (`.next/` from `next build`), reads the manifests under `distDir`, and reports per-route bundle sizes split into six file categories. Supports markdown (default) or JSON output, sorting, limiting, and per-category file listings. ### Why? We want a **static, build-output-based** way to compare different chunking strategies for JS and CSS without running the user's app. Existing tooling either requires running the app (`@next/bundle-analyzer` is a webpack plugin), is bundler-specific, or aggregates per-bundle/per-asset rather than per-route. This tool answers the concrete question "how much JS / CSS does each route ship today?" purely by reading static manifests, so it can be diffed across builds, branches, or bundlers (Turbopack vs webpack) and used to evaluate chunking changes. It is namespaced under `next internal` because the output format and category boundaries are tied to internal manifest shapes; we don't intend to make it a stable public surface yet. ### How? #### Command surface ``` Usage: next internal static-routes-info [directory] [options] Options: --json Output as JSON instead of markdown. --limit <n> Only show the first N routes after sorting (totals always reflect all routes). --sort <key> Sort routes by: name (default, ascending), or one of client, client-js, client-css, client-map, server, server-bundled-js, server-unbundled, server-map, total (descending). --files Include the list of files (relative to the output directory) per category in the JSON output. Requires --json. -h, --help Displays this message. ``` `--limit` and `--sort` always consider every route for the totals; only the displayed table is trimmed/reordered. `--files` is only meaningful in JSON output and errors otherwise. Invalid `--sort` keys error with the valid set listed. #### Six categories per route Each file the tool sees is placed into **exactly one** of these six buckets, so totals are not double-counted: | Category | Description | | ------------------ | -------------------------------------------------------------------------------------------------- | | Client JS | `.js` chunks loaded by the browser | | Client CSS | `.css` files loaded by the browser | | Client Source Maps | `.map` files for client JS / CSS | | Server Bundled JS | `.js` chunks executed on the server (App Router, Pages SSR, route handlers, middleware) | | Server Unbundled | Files traced via `*.nft.json` outside `distDir` (typically `node_modules` deps for `serverExternalPackages` / Pages SSR) | | Server Source Maps | `.map` files for server JS, including `.map`s referenced from nft.json | Source maps are discovered three ways: `.map` extension matches, `//# sourceMappingURL=...` trailers in JS, and `/*# sourceMappingURL=...*/` trailers in CSS. Maps always go into the `Maps` category even when the manifest puts them next to their bundle, so they never inflate Bundled or CSS counts. `sourceMappingURL` reads are memoized per chunk so a chunk shared by N routes is opened once. #### Two-step measurement 1. **Capture per-route file sets** by reading manifests: - `pages-manifest.json` and `app-path-routes-manifest.json` for the route list. - `<entry>.nft.json` for server-bundled chunks plus traced node_modules deps. - `<entry>_client-reference-manifest.js` for App Router client JS/CSS (both `entryJSFiles` / `entryCSSFiles` on Turbopack and `clientModules.chunks` on webpack — the parser handles both layouts). - `build-manifest.json` for shared App Router root chunks (`rootMainFiles`) and Pages Router client chunks. - `middleware-manifest.json` for middleware and edge route handlers. 2. **Deduplicate inside each per-route category, then measure.** A global `lstat` cache stats every unique path once across the whole run; per-category sets dedupe via string equality. Files are routed by extension (`.map` → Maps, `.css` → CSS, `.js` → bundled / unbundled depending on whether the path stays inside `distDir`) at the point they enter a set, so e.g. an `.nft.json` referencing both bundle and `.map` paths places each in the right bucket. #### Shared metric For each route, every category also carries a `sharedAvg`: the average size of the *intersection* between this route and each peer route of the same type. Computed as ``` sharedAvg = (Σ over peers p: |files(this) ∩ files(p)|) / number_of_peers ``` with both file count and bytes reported. The metric is also expressed as a percentage of the route's own count and bytes (`percentCount`, `percentBytes`) to make sharing easy to interpret at a glance — e.g. `5.3 files (88%) / 424.12 KB (100%)` means "88% of this route's files, and effectively all of its bytes, are also shipped by an average peer". Routes with no peers (only one route of their type) get `null`. Note that percentages are NOT commutative across peers (they're divided by each route's own count/bytes) while raw intersection numbers are. #### Output Markdown (default), with three sections — `## Routes`, `## Shared (avg per other route of same type)`, `## Totals` — each rendered as a fixed-width aligned table. Empty cells render as `-` (and routes with no peers in the Shared section as `n/a`) so meaningful values stand out: ``` ## Routes | Route | Type | Client JS | Client CSS | Client Source Maps | Server Bundled JS | Server Unbundled | Server Source Maps | | ------------ | ---------- | ------------------- | --------------- | ------------------ | -------------------- | ------------------- | ------------------- | | / | app-page | 6 files / 424.40 KB | 2 files / 153 B | - | 16 files / 384.47 KB | 140 files / 1.37 MB | 16 files / 2.31 MB | | /api/edge | app-route | - | - | - | 9 files / 296.82 KB | - | 4 files / 1.53 MB | … ## Shared (avg per other route of same type) | Route | Type | Client JS | Client CSS | Client Source Maps | Server Bundled JS | Server Unbundled | Server Source Maps | | ------------ | -------- | ---------------------------------- | ---------------------------- | ------------------ | -------------------------------- | --------------------------------- | ------------------------------ | | / | app-page | 5.3 files (88%) / 424.12 KB (100%) | 1.3 files (63%) / 52 B (34%) | - | 11 files (69%) / 357.52 KB (93%) | 140 files (100%) / 1.37 MB (100%) | 11 files (69%) / 2.19 MB (95%) | … ``` JSON has the same per-category structure (count + bytes + sharedAvg + optional files list when `--files` is used) with identical category ordering: `clientJs`, `clientCss`, `clientMaps`, `serverBundled`, `serverUnbundled`, `serverMaps`. JSON values are exact (e.g. `0/0` is preserved as `{count:0, bytes:0}` rather than `-`) so machine consumers aren't affected by the markdown placeholder. Totals also expose dedup'd `files` arrays under `--files`. #### Route types Reported types: `app-page`, `app-route`, `pages`, `pages-static`, `pages-api`, `middleware`. App Router route handlers with `runtime: 'edge'` report as `app-route` (not a separate `edge-function`) so they're directly comparable with their Node-runtime peers. Middleware is a first-class type rather than being lumped under edge-function. #### Robust manifest parsing `_client-reference-manifest.js` is a JS module, not JSON. Both bundlers emit it but with different layouts: - Turbopack: multi-line, with a `for (const key in MANIFEST[entry].clientModules) MANIFEST[entry].clientModules[k] = val` suffix when a deployment ID is set. - Webpack: single-line, no whitespace around `=`. We extract the JSON body without evaluating the file. The implementation locates the `globalThis.__RSC_MANIFEST[` anchor, walks the JS string literal that holds the entry name (honoring `\\` escapes), then balance-walks the `{...}` body. This handles entry names that contain `]` characters, e.g. ```js globalThis.__RSC_MANIFEST["/(dashboard)/[teamSlug]/(team)/~/stores/(store-details)/blob/[storeId]/page"] = {...} ``` Any structural surprise (anchor missing, unterminated string/object, JSON parse failure) throws with the file path and offset — we never silently undercount client JS/CSS for a route. Only file-not-found stays as a `null` return — that's a normal case for server entries with no client-reference manifest (middleware, route handlers, etc.). ### Tests `test/production/static-routes-info/` is a real fixture covering every route type: - App Router: `/`, `/about`, `/no-client`, `/items/[itemId]` (a dynamic segment inside a `(group)` route group, which forces `]` to appear unescaped in the manifest entry name and exercises the parser), plus the auto-generated `/_not-found`. - App Router route handlers: `/api/node` (default Node runtime), `/api/edge` (`runtime: 'edge'`). - Pages Router: `/pages-ssr`, `/pages-ssr-2` (siblings sharing chunks), `/pages-static`, `/api/hello`. - Middleware: `middleware.ts`. - A shared lib (`lib/shared.ts`) imported by both pages-router siblings and `/`-`/about` to give the shared-avg metric something non-trivial to measure. - A `'use client'` `Counter` component imported by `/` and `/about` (but not `/no-client`), which itself imports `counter.module.css`. Routes that import Counter must ship strictly more client JS (and on Turbopack, more client CSS) than `/no-client` — this is asserted, and it's the cross-bundler regression check for the App Router client-JS collection on webpack via `clientModules.chunks` (without it, every webpack app-page reports `clientJs.count = 0`). The test file (`static-routes-info.test.ts`) covers all the above plus output formats, sort options, limit semantics, file-list integrity, totals dedup, shared-avg correctness against a hand-computed reference, markdown/JSON consistency, and the empty-cell `-` placeholder. The shared-avg metric is verified three independent ways: against a from-scratch reimplementation that walks the `--files` lists and recomputes every (route, category) cell; against a "sharedAvg.count == own.count IFF every peer is a strict superset" invariant that makes 100% values load-bearing; and against a hand-known case where one route ships a chunk no peer does, forcing strictly-below-100% sharing. 31 tests, passing on both Turbopack and webpack. The tool was also exercised against `bench/basic-app`, `bench/heavy-npm-deps`, `bench/nested-deps`, `bench/app-router-server`, and `bench/nested-deps-app-router` while developing. ### Notes for reviewers - New error codes added to `errors.json` for the manifest-parser throws and other invariant violations. - The command is registered under `next internal`; not advertised in user-facing docs by design. - Webpack quirk documented in the test: `flight-manifest-plugin.ts`'s `mergeManifest` merges every app-page's `entryCSSFiles` into every other route's CRM, so per-route CSS attribution on webpack is inherently fuzzy — the test asserts CSS attribution on Turbopack only, and the comment explains why. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-12 18:59:51 +00:00
3 people	5d0dab5fc6	[turbopack] Enforce root attribute for strongly consistent reads and collectibles (#93114) ... ### What? Replaces the silent "make root node" promotion in the turbo-tasks backend with a panic that enforces tasks to already have the `root` attribute before performing strongly consistent reads, reading task collectibles, or removing collectibles. ### Why? Previously, the backend would silently promote any non-root task to a root node (aggregation number `u32::MAX`) when these operations were requested. This masked incorrect task configuration — tasks that needed root-level aggregation weren't explicitly declared as such, making it harder to reason about the aggregation graph and hiding potential performance issues from implicit promotions. ### How? **Backend enforcement (3 locations):** - **Strongly consistent read** (`backend/mod.rs`): Checks `NativeFunction.is_root` on the target task. If it's a persistent task without `root`, panics with both the target and reader task descriptions. - **Read task collectibles** (`backend/mod.rs`): Same check when reading collectibles from a task. - **Remove collectibles** (`operation/update_collectible.rs`): Same check when removing collectibles (count < 0). All three locations drop the task guard before panicking to avoid deadlocking with `debug_get_task_description`. **Macro support for `root` on methods:** - `value_impl_macro.rs`: Now reads the `root` attribute from `#[turbo_tasks::function(root)]` on inherent impl and trait impl methods (previously hardcoded to `false`). - `value_trait_macro.rs`: Same for trait default methods. **`root` attribute additions:** - All `#[turbo_tasks::function(operation)]` in test files, benchmarks, and fuzz code - Production operations in `crates/next-api/` that are read with strong consistency - Regular functions and methods in tests that use `.strongly_consistent()` - Trait methods in `value_impl` and `value_trait` blocks used with strong consistency **New test:** - `non_root_task_panic.rs`: Verifies the panic fires when attempting a strongly consistent read on a non-root operation task. Captures the panic from the worker thread via a panic hook since the panic propagates as a channel error to the test thread. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-05-12 18:56:02 +00:00
Tobias KoppersandGitHub	4bad5e74fe	Turbopack: fix lock-order inversion between Storage::map and Storage::snapshots (#93788) ... ### What? Fix a latent deadlock between `track_modification_internal` / `SnapshotShardIter::next` and `Storage::end_snapshot` in `turbo-tasks-backend`, and harden `dash_map_multi::RefMut` so that holding a `StorageWriteGuard` across an `.await` can no longer compile. ### Why? `Storage` has three sharded `DashMap`s — `task_cache`, `map`, and `snapshots`. The two call sites that touch both `map` and `snapshots` (`track_modification_internal` at `storage.rs:685/703` and `SnapshotShardIter::next` at `storage.rs:845/851`) hold a `map` shard write lock and then take a `snapshots` shard write lock — i.e. order `map → snapshots`. `end_snapshot` did the opposite: it called `parallel::for_each(self.snapshots.shards(), ...)`, took a `snapshots` shard write lock, drained it, and then called `self.map.get_mut(&key)` to promote flags — order `snapshots → map`. The existing comment argued this was safe because `track_modification` only inserts into `snapshots` when `snapshot_mode == true` and `end_snapshot` first stores `snapshot_mode = false`. That argument is incorrect. `track_modification_internal` loads `snapshot_mode` (line 616) and inserts into `snapshots` later (line 685 or 703); the two are not atomic. The interleaving: 1. T1 in `track_modification_internal`: holds `map[N]`, reads `snapshot_mode() → true`. 2. T2 in `end_snapshot`: stores `snapshot_mode = false`, takes `snapshots[N]` write lock. 3. T2: tries `map.get_mut(&key)` → blocks on `map[N]` (T1 holds it). 4. T1: tries `snapshots.insert(...)` → blocks on `snapshots[N]` (T2 holds it). deadlocks both threads. Separately, `dash_map_multi::RefMut` (which backs `StorageWriteGuard`) had `unsafe impl Send`. The wrapped `parking_lot` `RwLockWriteGuard` is intentionally `!Send` upstream (via `lock_api::GuardNoSend(*mut ())`); the manual override let callers compile code that holds a `StorageWriteGuard` across an `.await`. In that pattern the guard pins a shard write lock from a parked future and every other tokio worker piles up trying to take the same shard — a deadlock that the borrow checker would otherwise catch for free. ### How? **`Storage::end_snapshot`** (`storage.rs:383`): - Zip `map.shards()` with `snapshots.shards()` by index and lock each pair in the documented order: `map_shard.write()` first, then `snap_shard.write()`. - For every key drained from `snapshots[N]`, resolve it directly in the held `map_shard` guard via `RawTable::find`, rather than calling `self.map.get_mut` (which would re-enter the same shard). - `debug_assert_eq!` on the shard counts. - Update the obsolete "this is fine" comment to describe the prior race and the new pattern. **Shard pairing invariant** is now documented on the `map` and `snapshots` field declarations: both `DashMap`s are constructed with the same `shard_amount`, the same `TaskId` key type, and the same stateless `FxBuildHasher`, so shard `N` in `snapshots` corresponds exactly to shard `N` in `map`. Every key present in `snapshots.shards()[N]` (if present in `map`) lives in `map.shards()[N]`, so the zipped acquisition covers every drainable entry. **`dash_map_multi::RefMut`** (`dash_map_multi.rs:32`): - Removed the `unsafe impl<K: Eq + Hash + Sync, V: Sync> Send for RefMut<'_, K, V> {}`. - `RwLockWriteGuard` already contains `GuardNoSend(*mut ())`, so the auto-trait derivation correctly marks `RefMut` (and therefore `StorageWriteGuard`, `TaskGuardImpl`, `OccupiedEntry`, `VacantEntry`) `!Send`. Verified by compile-fail probes on all four guard types. - Kept `unsafe impl Sync` — sharing `&RefMut` between threads is still sound (the write guard provides exclusive access, `K: Sync + V: Sync`). - Replaced the old SAFETY comment with one that explains why `Send` is intentionally omitted. ### Audit summary After the fix, the full lock-order graph for the three `Storage` dashmaps is: ``` task_cache ─→ map ─→ snapshots ▲ └── (non-blocking try_lock_and_remove + defer in evict_after_snapshot) ``` Total order: `task_cache → map → snapshots`. The only reverse edge is `evict_after_snapshot`'s non-blocking `try_lock_and_remove` on `task_cache` while holding `map`, which defers on contention — no hold-and-wait possible. Within `map`, the only multi-guard path is `access_pair_mut`, whose retry loop in `get_multiple_mut` holds zero shard guards at every blocking acquire. Per-thread nested task guards are prevented at runtime in debug builds by `TaskLockCounter::acquire` (`operation/mod.rs:128–166`). ### Testing - `cargo check --all-targets` (turbopack workspace): clean. - `cargo test -p turbo-tasks-backend --lib storage::`: 4/4 pass, including `modify_during_snapshot_clears_live_modified_flags` and `modify_different_category_during_snapshot` which exercise the snapshot lifecycle this PR touches. - Compile-fail probes confirmed `StorageWriteGuard`, `TaskGuardImpl`, `OccupiedEntry`, and `VacantEntry` are all `!Send` post-change. <!-- NEXT_JS_LLM_PR -->	2026-05-12 18:36:38 +00:00
Tobias KoppersandGitHub	191fd7428b	turbopack: reschedule stale tasks with correct invalidation priority (#92897) ... ### What? When an in-progress task is invalidated during execution, it transitions to a "stale" state. Previously, on completion it was directly re-executed in the same worker slot — inheriting the original schedule priority rather than the priority from the invalidation that made it stale. ### Why? A stale task that was invalidated at low priority was being re-executed at whatever high priority the original schedule had. This caused high-priority work to be unfairly blocked or deprioritized in the scheduler. ### How? **`backend.rs` trait:** `task_execution_completed` return type changed from `bool` (reschedule yes/no) to `Option<TaskPriority>` — `None` means done, `Some(priority)` means the task was stale and must be re-executed at this priority. **`backend/mod.rs`:** The three helper functions (`_prepare`, `_connect`, `_finish`) and the main `task_execution_completed` all propagate the invalidation priority on stale returns. In each stale path, the priority is read from `task.is_dirty().unwrap_or(TaskPriority::leaf())` before the task state is mutated. **`manager.rs`:** The executor no longer loops to directly re-execute stale tasks. Instead, if `task_execution_completed` returns `Some(stale_priority)`, the task is unconditionally re-scheduled through the priority runner at that priority, so all tasks execute in the correct priority order. <!-- NEXT_JS_LLM_PR -->	2026-05-08 12:00:46 +00:00
Tobias KoppersandGitHub	91eb83169d	Trace-server: Fix bottom up, and reduce allocations in turbopack-trace-server bottom-up grouping (#93460) ... ### What? Speeds up the bottom-up grouping pass in `turbopack-trace-server` by removing per-span `RcStr` allocations and fixes the hash map using the correct hasher for the grouping `HashMap`. ### Why? When loading large traces, building the bottom-up graph spent measurable time (1) allocating fresh `RcStr` values from `&str` keys for every span just to use them as `HashMap` keys, and (2) hashing those keys with the default randomized hasher. Both are avoidable: the underlying spans already own `RcStr`s, and the `StringTupleRef` equivalence-based lookup needs `FxHasher` anyway because `RcStr`'s `Hash` impl only matches `&str`'s `Hash` under `FxHasher`. ### How? - Change `nice_name`/`group_name`/`args` accessors on `SpanRef` and friends to return `&RcStr` instead of `&str`. The bottom-up grouping code can now clone the existing `RcStr` (a cheap ref-count bump) instead of allocating a new one from a `&str`. - Switch the `(RcStr, RcStr) -> SpanBottomUpBuilder` map in `bottom_up.rs` to use `FxBuildHasher`. This is required for the `StringTupleRef` equivalence lookup to produce matching hashes for the owned `(RcStr, RcStr)` keys, and it also removes the overhead of the default randomized hasher. - Update the `string_tuple_ref` test to use `FxBuildHasher` accordingly. No behavior changes; this is a pure refactor for performance inside the trace viewer tool. <!-- NEXT_JS_LLM_PR -->	2026-05-08 06:19:16 +00:00
5 people	505771a20d	Use Next.js version as Turbopack persistent cache versioning key (#93605) ... ### What? The Turbopack persistent cache directory under `.next/cache/turbopack/<version>/` is now versioned by the Next.js package version concatenated with the git short SHA, instead of `git describe`. Example: `v16.0.1-canary.13-94e9fa6` ### Why? The previous version key came from `VERGEN_GIT_DESCRIBE` (`git describe --match 'v[0-9]' --dirty`), which depends on having an annotated `v*` tag reachable from the build commit. That ties the cache version to local git tag state and breaks down for builds against forks, shallow clones, or branches without a recent matching tag. The Next.js package version from `packages/next/package.json` is the value users actually care about for cache compatibility, and it is already plumbed through to the Rust side as `ProjectOptions::next_version` (sourced from `process.env.__NEXT_VERSION`). Combining it with the git short SHA keeps per-commit cache uniqueness while removing the dependency on git tags. ### How? **Rust (`crates/next-napi-bindings`)** - `build.rs`: also emit `VERGEN_GIT_SHA` (short form) via `vergen_gitcl`. `VERGEN_GIT_DESCRIBE` is still emitted and is still used for the bug-report URL / panic log. - `src/next_api/turbopack_ctx.rs`: - New `cache_describe(next_version)` returns `format!("v{next_version}-{}", env!("VERGEN_GIT_SHA"))`. - `git_version_info` now takes the pre-built `describe` string instead of reading `VERGEN_GIT_DESCRIBE` itself. `dirty` handling is unchanged (still derived from `VERGEN_GIT_DIRTY`, still suppressed in CI), so `handle_db_versioning`'s dirty-repo behavior is preserved. - `create_turbo_tasks` takes `next_version: &str` and threads it through. - `src/next_api/project.rs`: - `project_new` passes `options.next_version` into `create_turbo_tasks`. No new field is needed on `NapiProjectOptions` — the `nextVersion` field already exists and is already populated by all three call sites. - `turbopack_database_compact` (the napi `databaseCompact` function used by `next-post-build`) gains a `next_version: String` parameter so post-build compaction targets the same versioned directory. **JS (`packages/next`)** - `src/build/swc/types.ts`, `generated-native.d.ts`, `index.ts`: update the `databaseCompact(path, nextVersion)` signature on both the native and WASM-fallback paths. - `src/cli/next-post-build.ts`: passes `process.env.__NEXT_VERSION` through to `bindings.turbo.databaseCompact`. `turbopack/crates/turbopack-cli` (the standalone Turbopack CLI) is intentionally left on `VERGEN_GIT_DESCRIBE` — it is not tied to a Next.js package version. --------- Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-08 05:53:01 +00:00
5 people	b785969e83	Order loader tree imports by tree depth (#93537) ... ### What? Reorder the imports emitted by the app-page loader-tree builder so that they are grouped by their depth in the loader tree (shallow segments first, deep segments last) instead of the somewhat arbitrary order produced by recursive walking. ### Why? The loader tree is built by walking the app router tree recursively, and each segment contributes a few `require(...)` declarations (layout, error, loading, page, metadata, ...) to a shared list of imports that is later concatenated into the generated entry module. The order of that list is what ends up in the bundle, and walking the tree recursively interleaves segments at different depths in a way that depends on traversal order rather than on the structure of the route. For chunking and readability we want a more predictable order: *outer (shallower) layouts/segments before inner (deeper) ones*, with imports inside the same depth keeping their original relative order. This gives downstream chunking a more useful signal. ### How? In `crates/next-core/src/base_loader_tree.rs`: - `BaseLoaderTreeBuilder::imports` is now `Vec<(u32, RcStr)>` instead of `Vec<RcStr>`. The `u32` is a sort key (the depth at which the import was produced). - `create_module_tuple_code` takes a new `position: u32` argument and stores it alongside the generated `require` line. In `crates/next-core/src/app_page_loader_tree.rs`: - `walk_tree` takes a new `depth: u32` argument. The initial call from `build()` passes `0`. - `depth` is threaded through `write_modules_entry`, `write_metadata`, `write_metadata_items`, `write_metadata_item`, and `write_static_metadata_item`, so the three other places that push directly into `self.base.imports` (dynamic image metadata, static metadata items, and their alt-text companions) also tag their entries with the current depth. - When recursing into `parallel_routes`, depth is incremented only for the `"children"` slot. Named parallel routes (e.g. `@modal`) are sibling slots of the same segment and therefore stay at the same depth. - In `AppPageLoaderTreeBuilder::build`, the collected `(depth, import)` pairs are stable-sorted by depth (`sort_by_key`, which is stable in Rust, preserving the original relative order within each depth) and then stripped back to `Vec<RcStr>` before being placed on `AppPageLoaderTreeModule.imports`. The public type of `AppPageLoaderTreeModule.imports` (`Vec<RcStr>`) is unchanged, so consumers in `crates/next-core/src/next_app/app_page_entry.rs` need no adjustments. Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-07 10:37:54 +02:00
5 people	b7cc99695b	Turbopack: lazy aggregation optimize via persistent pending flag (#93454) ... ### What? Make the aggregation `optimize_queue` in the Turbopack persistent backend bounded and lazy. Cap the in-memory queue size, persist an `optimization_pending` flag per task, drain the queue per `process()` call with a per-queue lifetime budget, and recover dropped optimizations opportunistically via the flag instead of via an unbounded scheduler-side queue. ### Why? The previous implementation pushed an `OptimizeJob` for every `push_optimize_task` call into a single in-memory queue with no upper bound. On large workloads (or pathological aggregation churn), this queue could grow very large and cause the thread that scheduled the optimization to do unbounded work, regressing latency for the operations that triggered the schedule. The goals of this change: - Bound the worst-case work any single `process()` call does for optimizations (per-queue budget). - Bound the in-memory queue size so memory use is predictable. - Avoid losing optimizations: anything we drop must be eventually recovered. - Keep the common fast path cheap — no extra `Meta`-category guard acquisitions when the optimization flows through normally. ### How? Persist a new `optimization_pending` flag on `TaskStorage` (`storage_schema.rs`) and use it to drive lazy recovery in `aggregation_update.rs`: - `push_optimize_task` only enqueues an in-memory `OptimizeJob` if the queue is under `MAX_OPTIMIZE_QUEUE_SIZE` (10000) and the per-queue lifetime budget `MAX_OPTIMIZATIONS_PER_QUEUE` (1000) hasn't been exhausted. If we can't enqueue, we set `optimization_pending = true` on the task so a future operation that visits this task will re-discover and re-enqueue the optimization. - Every `AggregationUpdateJob` handler calls `check_optimization_pending` on the primary task(s) it touches, which re-enqueues the optimize job if the flag is set (and the queue/budget allow). - `process()` drains the `optimize_queue` one job at a time (preserving the original "root first" ordering), counting against the per-queue budget. Once the budget is exhausted, further `OptimizeJob`s in the queue are dropped and the flag is left set on those tasks (so they recover later). - `optimize_task` clears `optimization_pending` at entry so the recovery loop eventually settles. - The flag is **only** written on the drop path — the common case (enqueue → process normally) does not touch `optimization_pending`, so no `Meta`-category guard contention is added on the hot path. - `OptimizeJob` carries a best-effort `flag_already_set` snapshot so that when a job is dropped at process time and the snapshot says the flag was already set, we skip the redundant write entirely. Most jobs originating from `check_optimization_pending` (the recovery path) and `optimize_task`'s self-re-enqueue carry this hint. - `try_enqueue_optimize_job` is `#[must_use]` so the contract \"if this returns false, set the flag\" is enforced at the type level. `lock_and_mark_optimization_pending` is shared between `push_optimize_task_by_id` and the budget-exhausted drop branch. - `optimizations_executed` is intentionally persisted with the queue so that suspending and resuming the queue cannot reset the per-queue budget. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-06 07:25:56 +00:00
5 people	9a4e4e7f64	trace-server: shrink_to_fit LazySortedVec after sorting (#93361) ... ### What? In `turbopack-trace-server`, call `shrink_to_fit` on the inner storage of `LazySortedVec<T>` immediately after the one-shot sort that runs the first time the value is dereferenced. ### Why? `LazySortedVec` is built as a write-only `SmallVec<[T; 1]>` during trace ingestion (every `push` resets the inner `Once` so subsequent reads will resort), and is then "frozen" the first time it's `Deref`'d: the `Once::call_once` closure runs `sort()` once and the slice is handed out unchanged for the rest of the program's lifetime. Because the `SmallVec` grew via repeated `push`es, its capacity is typically a power-of-two well above its final length. After the first read, that overhead is dead memory we keep until the trace-server process exits. For traces with many spans, this adds up across all the `LazySortedVec`s on each span (e.g. the per-span event lists feeding the bottom-up / aggregated views), so the resident set of an idle trace-server is meaningfully larger than necessary. Since the `Once` guarantees the vector is never mutated again after the closure runs, the post-sort moment is the natural and only safe place to release that excess capacity. ### How? In the `Deref for LazySortedVec<T> where T: Ord` impl in `turbopack/crates/turbopack-trace-server/src/lazy_sorted_vec.rs`, the `once.call_once` closure now: 1. Reborrows the inner storage through the existing `UnsafeCell` pointer. 2. Calls `sort()` (unchanged behavior). 3. Calls `shrink_to_fit()` on the same `SmallVec`. The existing safety invariant — that the `&mut` reborrow inside `call_once` is exclusive because `Deref` is the only code path that touches the inner cell without a `&mut self`, and `Once` guarantees the closure runs at most once before any reader observes the slice — covers `shrink_to_fit` for free; it's just another exclusive mutation in the same scope as the sort. The `Deref` return path is unchanged: callers still receive a `&[T]` over the now-sorted, now-fitted backing storage. No public API change. No behavior change for callers. No new dependencies. ### Verification - `cargo build -p turbopack-trace-server` - `cargo clippy -p turbopack-trace-server` - `cargo fmt` No new tests: this is a pure memory-footprint optimization on a path that is already exercised by every existing read of a `LazySortedVec`. <!-- NEXT_JS_LLM_PR --> Co-authored-by: v-work-app[bot] <262237222+v-work-app[bot]@users.noreply.github.com> Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-05-01 07:00:01 +02:00
4 people	292a3fba32	Report OS memory pressure in TurboMalloc and trace samples (#93333) ... ### What? Adds a new `TurboMalloc::memory_pressure()` method that returns a normalized OS-level memory pressure value in the range `0..=100` as `Option<u8>`. That value is attached to every memory sample in the tracing layer (`TraceRow::MemorySample`) and propagated through the trace-server so that span queries return a `memory_pressure_samples` vector next to the existing `memory_samples`. ### Why? Our current tracing only records the in-process allocator usage (`TurboMalloc::memory_usage()`), which does not tell us when the *operating system* is actually under memory pressure. We want that signal in the trace output to: 1. Surface real OS memory pressure in trace dashboards alongside our own allocation totals. 2. Eventually use it as input to task-eviction decisions in `turbo-tasks` (see branch description). This PR lands the plumbing; the eviction heuristic is not part of this change. ### How? **`TurboMalloc::memory_pressure() -> Option<u8>`** — new, in `turbopack/crates/turbo-tasks-malloc/`. Values are normalized so that `0` = no pressure, `100` = maximum pressure. Platform-specific backends: | Platform | Source | Notes | |---|---|---| | Linux | `/proc/pressure/memory` (`some` `avg10`), fallback to `(MemTotal - MemAvailable) / MemTotal` from `/proc/meminfo` | PSI is not available on all kernels (< 4.20, without `CONFIG_PSI`, restricted containers). The meminfo fallback keeps the signal meaningful on any standard Linux system and matches the semantics of Windows' `dwMemoryLoad`. | | macOS | `kern.memorystatus_level` sysctl (% free memory) | Pressure = `100 - level`, read via `libc::sysctlbyname`. | | Windows | `MEMORYSTATUSEX::dwMemoryLoad` via `GlobalMemoryStatusEx` (`windows-sys`) | Already a 0–100 percentage of physical memory in use. | | Other / wasm | — | Returns `None`. | All runtime failures (missing file, sysctl error, failed API call, unparseable content) silently yield `None` rather than panicking. **Wiring into tracing:** - `TraceRow::MemorySample` gains a `memory_pressure: u8` field. `0` is used when `memory_pressure()` returns `None` on unsupported platforms. - `RawTraceLayer::maybe_report_memory_sample` populates it on every sample (sampling cadence unchanged). - This is a breaking change to the postcard wire format of `MemorySample`; old trace files cannot be read by the new `turbopack-trace-server`. Given the dev-only nature of this data that seemed acceptable — let me know if a migration is desired. **Wiring into the trace-server:** - `Store::memory_samples` is now `Vec<(Timestamp, u64, u8)>` and `add_memory_sample(ts, memory, memory_pressure)`. - A new `Store::memory_pressure_samples_for_range(start, end) -> Vec<u8>` mirrors `memory_samples_for_range`: same `MAX_MEMORY_SAMPLES = 200` cap and same group-and-max downsampling, so both vectors align index-by-index for a given span query. - `ServerToClientMessage::QueryResult` gains a `memory_pressure_samples: Vec<u8>` field next to `memory_samples`. **Dependencies:** `libc` (macOS only, `cfg`-gated), `windows-sys` with the `Win32_System_SystemInformation` feature (Windows only, `cfg`-gated). No new deps on Linux. ### Verification - `cargo build -p turbo-tasks-malloc -p turbopack-trace-utils -p turbopack-trace-server` - `cargo clippy -p turbo-tasks-malloc -p turbopack-trace-utils -p turbopack-trace-server --all-targets -- -D warnings` - `cargo test -p turbo-tasks-malloc` — 7 tests pass, including: - `memory_pressure_is_in_range`: asserts `Some(_)` and `≤ 100` on Linux, macOS and Windows (via `cfg`-gated `.expect()`), and allows `None` elsewhere. - Parser tests for both PSI and `/proc/meminfo` code paths (typical content, malformed input, clamping). - Runtime sanity check on the Linux sandbox: `/proc/pressure/memory` is absent (kernel 5.10 without `CONFIG_PSI`); the `/proc/meminfo` fallback returned `Some(3)` as expected. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-29 11:39:23 +02:00
Tobias KoppersandGitHub	99bbaa00c4	[rust-deps] Update tokio (#93195) ... ## What? Updates the tokio and tokio-util dependencies to their latest versions: - `tokio`: `~1.47.3` → `1.52.1` - `tokio-util`: `0.7.13` → `0.7.18` ## Why? Keep Rust dependencies up-to-date to benefit from bug fixes, performance improvements, and security patches. ## How? Updated version constraints in `Cargo.toml` and regenerated `Cargo.lock`. <!-- NEXT_JS_LLM_PR -->	2026-04-28 16:09:31 -07:00
5 people	dab281a3ab	Fix export * as X from './self' in scope-hoisted modules (#93192) ... ### What? Scope-hoisted execution of a module that re-exports its own namespace (`export * as X from './self'`) returned the wrong namespace for named imports of the module. Given: ```js // data.js import * as Self from './data' export function foo() { return 'foo' } export function bar() { return 'bar' } export function fooViaSelf() { return Self.foo() } // Self.foo undefined export * as Data from './data' ``` Any binding imported from `./data` that relied on the module's own namespace (e.g. `Self.foo`, `Data.foo`, `Data.Data.foo`) was `undefined` at runtime. Without scope hoisting the same code worked correctly. This PR fixes the bug and adds execution tests covering self-namespace re-exports — with and without scope hoisting — including chained re-exports (`Data.Data.foo`, `Data.Data.Data.bar`). ### Why? For an access like `Self.Data` where `Self = import * as Self from './data'` and `Data` is exposed through `export * as Data from './data'`, the namespace-member-access optimization rewrites the reference to a named import resolving to a synthesized rename module (`./data <export * as Data>`). `ReferencedAsset::get_ident_inner` then recurses through the rename's `EsmExport::ImportedNamespace("Data")` and returns a `namespace_ident` derived from the inner module's chunk-item id — but `EsmAssetReference::code_generation` independently took `id = referenced_asset.chunk_item_id` and emitted ```js var <inner-data-ident> = __turbopack_context__.i("<rename-id>"); ``` so the variable named like `ns(data.js)` actually held `ns(rename) = { Data: ns(data.js) }`. The non-optimized `import * as Self` uses the same mangled name and sees the rename's namespace, so `Self.foo()` evaluates to `undefined`. ### How? Keep the variable name and the `.i(...)` argument consistent by moving the "what to import" decision onto the ident itself: - `ReferencedAssetIdent::Module` gains an `import_source: ImportSource` field that describes what to import to populate the namespace variable. - `ImportSource` is an enum: - `Module { asset }` — carries a reference to the final module in any re-export chain, from which the chunk-item id is lazily computed. - `External { request, ty }` — carries everything needed to emit `__turbopack_external_import` / `__turbopack_external_require`. - The `namespace_ident` is cached in `ReferencedAssetIdent::Module` at resolution time (computed via `ImportSource::get_namespace_ident()`) so downstream sync visitors can read it without re-entering the async layer. - `ReferencedAsset::get_ident` / `get_ident_inner` populate the field. For in-group re-exports the inner module propagates up; for external references the `External` variant is used. - `EsmAssetReference::code_generation` destructures `ReferencedAssetIdent::Module { namespace_ident, ctxt, import_source, .. }` and dispatches purely on `import_source`; it no longer reads `referenced_asset` after the `get_ident` call. The hoisted-statement dedup key still uses the directly-referenced asset's id, so two references that happen to resolve to the same inner module via different paths (e.g. direct vs. through a rename) still emit separate `var` declarations for AST merging to rename. - ESM-external gating (`__turbopack_external_import` vs. `__turbopack_external_require`) stays where it was — the emit site reads `self.import_externals` from the surrounding `EsmAssetReference`, so `ImportSource::External` does not carry it. No additional `MergeableModuleExposure` or `additional_ids` changes are needed. The rename module is never referenced at runtime; no snapshot files change. ### Tests - `turbopack/crates/turbopack-tests/tests/execution/turbopack/exports/self-reexport-star/` — scope-hoisted execution test covering self-namespace re-exports, nested access (`Data.Data.foo`, `Data.Data.Data.bar`), chained re-exports through another module, and namespace key enumeration. - `turbopack/crates/turbopack-tests/tests/execution/turbopack/exports/self-reexport-star-no-hoisting/` — same test cases, run with scope hoisting disabled, reusing the fixtures from the sibling directory. Verified by `cargo test --test execution` (213 passed) and `cargo test --test snapshot` (89 passed) in `turbopack-tests`. No snapshot files are modified. Closes NEXT- Fixes # <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Niklas Mischkulnig <[email protected]>	2026-04-28 21:48:32 +02:00
3 people	e4a5b11b97	[turbo-tasks-backend] Single-item optimization for lost follower operations (#93200) ... ## What? Adds performance optimizations for the common case of single-item lost follower operations in the turbo-tasks-backend aggregation update system, along with improved tracing capabilities for debugging. ## Why? When profiling turbopack operations, we observed that many aggregation update jobs involve only a single upper losing a single follower. The previous implementation always used vectors and iteration even for these single-item cases, adding unnecessary allocation overhead. Additionally, the existing `trace_aggregation_update` feature was incomplete (compilation errors) and lacked visibility into task data and operation statistics. ## How? ### Single-item optimization - Added a dedicated `InnerOfUpperLostFollower` job variant and `inner_of_upper_lost_follower` function to handle the single upper + single follower case without vector allocation - Optimized the plural variants (`InnerOfUppersLostFollowers`, `InnerOfUppersLostFollower`, `InnerOfUpperLostFollowers`) to delegate to the singular function when they contain only one item ### Tracing improvements - Added `trace_aggregation_update_stats` feature flag that enables counters for all aggregation update job types (new_followers, lost_followers, balance_edge, optimize_task, etc.) recorded in trace spans - Fixed `trace_aggregation_update` feature compilation errors by using `task.get_task_description()` instead of `ctx.get_task_description(task_id)` - Gated the `TaskDataCategory` used in aggregation update `ctx.task(...)` calls behind a const that resolves to `Meta` by default and `All` when tracing is enabled, so traces can see full task data without affecting default performance <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Claude Opus 4.5 <[email protected]>	2026-04-28 21:45:43 +02:00
3 people	7b01c98a4a	[turbopack-trace-server] Performance improvements for span event handling (#93179) ... ## What? Performance improvements for the turbopack-trace-server, optimizing how span events are stored, sorted, and retrieved. ## Why? When processing large traces, the trace server was spending significant time on: 1. Repeated sorting of events for each render in ExecutionOrder mode 2. Recomputing corrected self time on repeated lookups 3. Memory pressure from very large traces ## How? ### Lazy Sorting with LazySortedVec Introduces a new `LazySortedVec<T>` data structure that stores events unsorted and defers sorting until first read via `Deref`. Uses `UnsafeCell` + `Once` for thread-safe one-time sorting. This avoids repeated sorting overhead during trace ingestion. ### Pre-sorted Span Events by Start Time - `SpanEvent::Child` now stores its `start: Timestamp` alongside the index - `SpanEvent` implements `Ord` to sort by start time (with SelfTime before Child for equal timestamps) - The viewer's `ExecutionOrder` mode no longer needs to sort - events are already in order ### Cached Corrected Self Time - `SpanEvent::SelfTime` now wraps a `SpanEventSelfTime` struct containing an `OnceLock<Timestamp>` for `corrected_self_time` - The expensive tree lookup is performed once and cached for subsequent accesses - `SpanEventSelfTimeRef` provides access to the cached value via `corrected_self_time()` ### DROP_SPANS Environment Variable Adds `DROP_SPANS=<count>` env var to skip the first N spans when loading traces: - Useful for reducing memory usage with very large traces - Tracks dropped span IDs to also skip their subsequent events (End, SelfTime, etc.) - Stats output shows dropped span progress (e.g., "1000 spans, 500/1000 dropped") <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-27 17:19:26 +02:00
4 people	44722b808c	Turbopack: import.meta.glob docs + edge case support (#92729) ... ## What? Adds documentation and TypeScript types for `import.meta.glob` in Turbopack, and implements additional edge cases for Vite compatibility. ### Documentation - New `## import.meta.glob` section in the [Turbopack reference page](/docs/app/api-reference/turbopack) covering: lazy/eager loading, named imports, query strings (including the object form), multiple patterns, negation, TypeScript types, and a full options table. - `import.meta.glob` row added to the Module Resolution feature table. - Migration note added to the [from-Vite guide](/docs/app/guides/migrating/from-vite) explaining that `import.meta.glob` works out of the box with Turbopack, with before/after for the deprecated `as` → `query` migration. ### TypeScript types - Added `ImportMetaGlobOptions` interface and overloaded `glob()` signatures to `ImportMeta` in `packages/next/types/global.d.ts`. - The overloads return `Record<string, unknown>` when `eager: true` is passed and `Record<string, () => Promise<unknown>>` otherwise. ### Implementation (edge cases) - **`import: '*'`** (namespace import): Treated the same as omitting the `import` option (returns the whole module namespace), matching Vite semantics. Previously would have generated broken `module["*"]` access. - **`query` as object literal**: `{ query: { bar: 'foo', raw: true } }` is now supported. Keys and values are URL-encoded and joined into a query string (`?bar=foo&raw=true`). - **Stricter option validation**: A non-object-literal second argument (e.g. `import.meta.glob('./*.js', 'eager')`) is now a compile-time error instead of a warning, since the options cannot be safely defaulted. ### Tests - Execution tests for new features: namespace `import: '*'`, query object, combining query with negation, explicit dotfile patterns. - New `import-meta-glob-errors` execution test with issue snapshots covering: too many arguments, non-string pattern, and non-object options argument. ## Why? 1. The feature shipped in #92640 but had no user-facing documentation and no TypeScript types. 2. Several Vite-compatible edge cases were not handled: namespace imports (`import: '*'`) and query objects. These gaps would cause subtle behavioral differences for users migrating from Vite. 3. Option validation was too lenient — invalid second arguments fell through to defaults silently. ## How? ### Docs & types - `docs/01-app/03-api-reference/08-turbopack.mdx` — new section, table row, TypeScript subsection - `docs/01-app/02-guides/migrating/from-vite.mdx` — migration note - `packages/next/types/global.d.ts` — `ImportMetaGlobOptions` + `ImportMeta.glob()` overloads ### Implementation - `turbopack/crates/turbopack-ecmascript/src/references/import_meta_glob.rs`: - `parse_import_meta_glob()`: `import: '*'` normalized to `None`; query object parsing via `urlencoding::encode()`; non-object options argument now emits an error and returns `None`. - `turbopack/crates/turbopack-ecmascript/Cargo.toml`: added `urlencoding` workspace dep. Note: dotfile/dot-directory filtering is not needed in `import.meta.glob` itself — the underlying `Glob` pattern engine and `read_glob_internal` already handle dotfile semantics correctly (explicit `./.foo/*.js` matches, wildcards include them). ### Tests - `turbopack-tests/tests/execution/turbopack/resolving/import-meta-glob/` — new cases for `import: '*'`, query object, query + negation, explicit dotfile pattern, wildcard including dotfiles - `turbopack-tests/tests/execution/turbopack/resolving/import-meta-glob-errors/` — new test suite for fatal parse errors with issue snapshots ### Checklist - [x] `pnpm prettier-fix` / `cargo fmt` run - [x] Documentation follows the [docs contribution guide](https://nextjs.org/docs/community/contribution-guide) - [x] Execution tests pass (`cargo test -p turbopack-tests --test execution -- "import_meta_glob"`) - [x] `cargo clippy` clean - [x] `pnpm --filter=next types` passes <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-27 07:37:53 +00:00
Tobias KoppersandGitHub	e7bcba89dc	[rust-deps] Update bytes, crc32fast, memchr, pin-project-lite, quote, rand, regex, ringmap, roaring (#93197) ... ### What? Updates Rust dependencies to their latest versions: | Crate | From | To | |-------|------|-----| | bytes | 1.1.0 | 1.11.1 | | crc32fast | 1.4.2 | 1.5.0 | | memchr | 2.7.4 | 2.8.0 | | pin-project-lite | 0.2.9 | 0.2.17 | | quote | 1.0.23 | 1.0.45 | | rand | 0.10.0 | 0.10.1 | | regex | 1.10.6 | 1.12.3 | | ringmap | 0.2.3 | 0.2.5 | | roaring | 0.10.10 | 0.11.4 | ### Why? Keep dependencies up to date with latest bug fixes, performance improvements, and security patches. ### How? Version bumps in `Cargo.toml` with corresponding `Cargo.lock` updates. <!-- NEXT_JS_LLM_PR -->	2026-04-27 09:33:35 +02:00
Tobias KoppersandGitHub	5d3232fb2b	[rust-deps] Update tracing (#93194) ... ### What? Updates the `tracing` crate and its related dependencies to their latest versions: - `tracing`: 0.1.37 → 0.1.44 - `tracing-attributes`: 0.1.30 → 0.1.31 - `tracing-core`: 0.1.34 → 0.1.36 ### Why? Keep Rust dependencies up to date with security patches and performance improvements. ### How? Updated the version specification in `Cargo.toml` and regenerated `Cargo.lock`. <!-- NEXT_JS_LLM_PR -->	2026-04-24 14:24:36 +00:00
Tobias KoppersandGitHub	1e4782ad7b	[rust-deps] Update serde (#93196) ... ### What? Updates the serde family of crates to their latest versions: - `serde`: 1.0.217 → 1.0.228 - `serde_json`: 1.0.138 → 1.0.149 - `serde_path_to_error`: 0.1.16 → 0.1.20 - `serde_qs`: 0.13.0 → 1.1.1 - `serde_with`: 3.12.0 → 3.18.0 ### Why? Keep dependencies up to date with bug fixes and performance improvements. ### How? Updated version constraints in `Cargo.toml` and ran `cargo update` to regenerate the lockfile. <!-- NEXT_JS_LLM_PR -->	2026-04-24 16:04:42 +02:00
4 people	b91cff03ad	feat(turbopack): add NEXT_TURBOPACK_TRACING_PATH to control trace output location (#93084) ... ### What? Adds a new environment variable `NEXT_TURBOPACK_TRACING_PATH` that lets users specify the full output path (including filename) for the Turbopack trace file. ### Why? Some users need to control where the `trace-turbopack` file is written — for example, to redirect it outside the project directory, to a shared volume, or to a custom filename. Previously the path was hardcoded to `<project>/.next-profiles/trace-turbopack` with no override mechanism. ### How? When `NEXT_TURBOPACK_TRACING` is set (tracing is enabled) and `NEXT_TURBOPACK_TRACING_PATH` is also set, the trace is written to the specified path instead of the default: - **Absolute paths** are used as-is. - **Relative paths** are resolved against the process CWD. - The **parent directory is auto-created** recursively (equivalent to `mkdir -p`), so pointing to a non-existent subdirectory works without manual setup. - Setting `NEXT_TURBOPACK_TRACING_PATH` alone (without `NEXT_TURBOPACK_TRACING`) has no effect — tracing is not enabled by it. - The "Trace output will be written to …" message reflects the resolved path. The change is confined to `crates/next-napi-bindings/src/next_api/project.rs`. **Example:** ```sh NEXT_TURBOPACK_TRACING=overview \ NEXT_TURBOPACK_TRACING_PATH=/tmp/my-project-trace \ next dev ``` <!-- NEXT_JS_LLM_PR --> Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-24 12:17:01 +02:00
3 people	08da80a777	[turbopack] Fix max_level_hint to return most verbose per-target level (#93128) ... ### What? Fix `FilterLayer::max_level_hint` in `turbopack-trace-utils` to return the most verbose `LevelFilter` across per-target configs instead of the least verbose one. ### Why? `tracing`'s `Subscriber` uses `max_level_hint` to short-circuit event dispatch globally: any event whose level is more verbose than the hint is dropped before `enabled()` is consulted. The previous implementation returned `self.config.values().copied().min()`, i.e. the *least* verbose configured level. As soon as any target was filtered to a low verbosity (e.g. `WARN`), events at more verbose levels (`INFO`/`DEBUG`/`TRACE`) were discarded for *all* targets, even those whose own per-target config would have allowed them. ### How? Swap `min()` → `max()` in `FilterLayer::max_level_hint` so the hint reflects the most verbose level any configured target could enable. Per-target filtering in `enabled()` is unchanged and continues to enforce the correct level for each target. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>	2026-04-24 11:08:09 +02:00
4 people	bb2a4457b6	CI: Track SWC native binary size in PR stats comment (#92938) ... ### What? Adds a new "Native Binary" section to the `Stats from current PR` comment posted by the `generate-pull-request-stats` GitHub Action. The section reports the size of the SWC native binary (`packages/next-swc/native/*.node`) built for the PR and the change relative to the latest value recorded on `canary`. Example row: > | Metric | Canary | PR | Change | Trend | > |:-------|-------:|---:|-------:|:-----:| > | SWC Binary Size | 113 MB | 121 MB | 🔴 +7.34 MB (+6%) | ▁▃▅▇█ | ### Why? The `.node` binary is a sizable, user-facing artifact (it ships with `next`). Today there's no visibility on its size in PRs, so regressions from Rust/Cargo changes are easy to miss. Surfacing the size (and a diff vs. canary) in the existing stats comment gives immediate feedback on any change that affects the binary. ### How? - **Measurement (`.github/actions/next-stats-action/src/run/index.js`):** After the action copies the pre-built native binary into the working dirs, sum the size of every `*.node` file under the action's `native/` directory and store it as `General.swcBinarySize` on both the canary and PR stat sets. - **Canary baseline via KV history (`.github/actions/next-stats-action/src/add-comment.js`):** The workflow downloads a single pre-built binary and copies it into both the canary and PR checkouts, so an in-run diff would always be zero. Instead, for PR runs we override `mainRepoStats.General.swcBinarySize` with the most recent `swcBinarySize` from the Vercel KV history (the same store already used for other canary metrics). Canary runs keep the measured value, and the existing persistence path (`saveToHistory` / `General`) writes it back so future PRs see an updated baseline. If no history exists yet, the Canary column renders `N/A` and no change is shown. - **Rendering:** New `generateNativeBinarySection` emits a `<details>`-wrapped section styled like Bundle Sizes / All Metrics, rendered right after the Bundle Sizes block. It reuses `prettify`, `formatChange`, and `generateTrendBar` so formatting, thresholds, and sparklines are consistent with the rest of the comment. A tight threshold entry (`swcBinarySize: { absoluteMin: 10 KB, percentMin: 0.5%, percentOnly: 0.05% }`) is added so meaningful movement is flagged while tiny determinism noise is ignored. - **Top-line summary:** Because `swcBinarySize` lives on `General`, the existing `generateChangeSummary` automatically promotes significant changes into the headline regression/improvement table at the top of the comment, using the label `SWC Binary Size`. - **Sharded runs:** The workflow runs two sharded jobs (Webpack and Turbopack). The binary is bundler-independent, so both shards report the same value; `aggregate-results.js` merges `General` via `Object.assign`, which is correct here (last-writer-wins on an identical value). No workflow changes are required — the existing `pull_request_stats.yml` already downloads the `next-swc-binary` artifact and copies it into the action's `native/` directory. ### Verification Smoke-tested `add-comment.js` locally with `LOCAL_STATS=1` in two scenarios: - No KV history configured → Canary column renders `N/A`, PR value renders, Change is `-`, section still shown. - Mocked KV history containing prior `swcBinarySize` values → full diff rendered in both the Native Binary section and the top-level significance summary, with a sparkline of the last 5 historical entries. Closes NEXT- <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-18 20:58:47 +02:00
Tobias KoppersandGitHub	87e7b083b8	Improve "make root node" trace attributes (#92941) ... ### What? Improve the trace span attributes for "make root node" operations in the turbo-tasks backend. Instead of recording only the raw `task_id`, the spans now include a human-readable task description (type name + task ID) via `debug_get_task_description`. ### Why? When profiling or debugging with tracing tools, raw task IDs are opaque and require additional lookups to understand which task is being processed. Including a human-readable task description (e.g. `TaskId(42) MyFunction(args...)`) makes traces immediately actionable. Two spans were affected: - `"make root node for strongly consistent read"` in `backend/mod.rs` - `"make root node for removing collectible"` in `backend/operation/update_collectible.rs` ### How? - Added `debug_get_task_description` method to the `ExecuteContext` trait in `operation/mod.rs` - Implemented it on `ExecuteContextImpl` by delegating to the existing `backend.debug_get_task_description` method (already used in other tracing spans) - Replaced `%task_id` (which formats only the ID) with `task = self.debug_get_task_description(task_id)` / `task = ctx.debug_get_task_description(task_id)` in both span definitions <!-- NEXT_JS_LLM_PR -->	2026-04-18 20:56:49 +02:00
4 people	3d3def23a1	Turbopack: optimize SelfTimeTree performance, memory usage, and fix range query boundaries (#92931) ... ### What? Optimize the `SelfTimeTree` data structure in the Turbopack trace server for better performance and lower memory usage when loading large trace files, and fix a correctness bug in range queries. ### Why? Two issues were identified: 1. **Performance**: The tree was splitting and rebalancing on every insert during bulk loading, and each `distribute_entries` call recursively triggered further split checks per entry, creating significant overhead. 2. **Correctness**: Range query boundary conditions used exclusive comparisons (`<`/`>`), incorrectly excluding entries that touched the exact start or end timestamp of the queried range. ### How? **Performance optimizations:** 1. **Batch split checks**: Add `insert_without_check` for inserting entries without triggering a split check per entry. In `distribute_entries`, entries are moved to children using `insert_without_check`, and `check_for_split` is called once per child after all entries are distributed (rather than once per entry). 2. **Lazy optimization with memory reclamation**: Add an `optimize()` method and a `for_each_in_range_optimize()` traversal method that distribute and rebalance nodes lazily: - `store.optimize()` is called when the reader reaches end-of-data (before waiting for more), doing a bulk pass to properly structure the tree after initial load. After distributing entries to children, each node calls `shrink_to_fit()` on its entries vec to release excess capacity. - `for_each_in_range_optimize()` distributes and rebalances each node it visits, so query paths are progressively optimized during normal use. 3. **Fix `rebalance()` bug**: `check_for_split()` was incorrectly called after merging subtrees during rebalance — replaced with `distribute_entries()` since the node already has children and needs redistribution, not a fresh split. **Correctness fix:** All range boundary comparisons in `for_each_in_range`, `for_each_in_range_optimize`, and `lookup_range_count` updated from exclusive (`<`/`>`) to inclusive (`<=`/`>=`), ensuring entries with timestamps exactly equal to the range endpoints are included in results. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-17 14:36:27 +02:00
4 people	e2e8ba9aca	turbo-trace-server: reduce allocation overhead when loading large trace files (#92896) ... ### What? Reduces allocation overhead in `turbo-trace-server` when reading large trace files. Measured improvement: **114 MB/s → 156 MB/s** (~37% throughput increase) on a 16 GB trace with 46 million spans. ### Why? Profiling showed the reader thread spending significant time in heap allocation and deallocation during the initial bulk-load phase. The hot spots were: 1. `to_string()` / `into_owned()` in `process_internal_row_queue` allocating a new `String` for every span name, category, and arg key/value — even when the same string (e.g. `"turbo_tasks::function"`) appeared in millions of spans. 2. `Vec<SelfTimeEntry>::push` growing one-at-a-time, triggering many `realloc` calls. 3. `OnceLock<Box<SpanTimeData>>` — inherent to the lazy-init design; mitigated by the allocator switch. 4. `Vec<TraceRow>::push` in the deserialization loop re-growing during the initial read. 5. `process_internal_row` allocating a fresh `Vec` queue on every call (one per row). 6. `Vec<Span>::push` and hash map rehashing in `Store::add_span`. 7. `into_static()` cloning forcing borrowed `Cow` strings to heap for queued rows. ### How? **1. mimalloc as global allocator (`main.rs`, `Cargo.toml`)** Added `turbo-tasks-malloc` as a dependency with a default-on `custom_allocator` feature (matching the pattern in `turbopack-cli` and `turbopack-nft`). Set `#[global_allocator] static ALLOC: TurboMalloc = TurboMalloc`. Also wired `TurboMalloc::thread_stop()` via rayon's `exit_handler` so each worker thread returns its mimalloc thread-local heap to the global pool on exit. **2. `RcStrInterning` in `turbo-rcstr`** New `RcStrInterning` struct backed by `FxHashSet<RcStr>`. Methods: `intern(&str)`, `intern_cow(Cow<str>)`, `intern_display(&Display)`. Strings shorter than the inline threshold (7 bytes on 64-bit) are already zero-allocation inline atoms and skip the set entirely. **3. `String` → `RcStr` for span fields** Changed `Span.name`, `Span.category`, and `Span.args` key/value types from `String`/`Vec<(String, String)>` to `RcStr`/`Vec<(RcStr, RcStr)>`. Same for `SpanNames` and `SpanBottomUpBuilder`. All `to_string()` / `into_owned()` calls in `process_internal_row_queue` are replaced with `interner.intern()` / `intern_cow()` / `intern_display()` calls on a per-`TurbopackFormat` `RcStrInterning`. Repeated names like `"turbo_tasks::function"` now share a single allocation via refcount bump instead of a fresh heap allocation each time. The `into_static()` path for queued rows also benefits: cloning an already-interned `RcStr` is a refcount increment, not a string copy. **4. Capacity hints** Pre-allocated major collections to avoid repeated reallocation during the initial bulk load: - `Store::spans`: `Vec::with_capacity(131_072)` - `TurbopackFormat::id_mapping`: `with_capacity(131_072)` - `TurbopackFormat::outdated_spans`: `with_capacity(8_192)` - `TurbopackFormat::queued_rows`: `with_capacity(1_024)` - `TurbopackFormat::thread_stacks`, `thread_allocation_counters`: `with_capacity(64)` - `TurbopackFormat::self_time_started`: `with_capacity(256)` **5. Reused queue Vec (hot spot #8)** `TurbopackFormat` now holds `row_queue: Vec<InternalRow<'static>>` as a field. `process_internal_row` takes it via `take` and puts it back after the loop, retaining the allocated capacity across calls instead of allocating a new `Vec::new()` for every row processed. **6. `TraceFormat::create_reused()` hook (hot spot #4)** Added an overridable `create_reused()` method to the `TraceFormat` trait. `TurbopackFormat` overrides it to return `Vec::with_capacity(4_096)`, giving the row deserialization buffer reasonable initial capacity to avoid repeated doubling during the initial bulk read. **7. `SelfTimeTree` leaf capacity (hot spot #2)** New child nodes created at split time get `Vec::with_capacity(SPLIT_COUNT / 2)` = 64 entries, matching the expected post-split occupancy and avoiding incremental doubling after every split. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-17 13:39:04 +02:00
4 people	1de67d8b34	Refactor trace server sort to SortMode enum (execution order / value / name) (#92873) ... ### What? Replaces the boolean `sort` flag in the trace server with a `SortMode` enum supporting three modes: - **`ExecutionOrder`** — no sorting, spans appear in natural/execution order (previous `false`) - **`Value`** — sort by corrected duration descending (previous `true`) - **`Name`** — sort alphabetically by span name (title), then by category ### Why? The boolean `sort` only allowed toggling duration-based sorting on/off. Adding alphabetical sorting by name makes it easier to find duplicate spans and locate a specific span by name in the trace viewer. The enum is extensible for future sort modes without breaking changes. ### How? **Rust (`turbopack-trace-server`):** - Added `SortMode { ExecutionOrder, Value, Name }` enum to both `lib.rs` (for the query API) and `viewer.rs` (for the WebSocket viewer). - `ViewMode` variants changed from `sorted: bool` to `sort_mode: SortMode`. - `sort_children()` now returns `SortMode` instead of `bool`; all 10 sorting branches in `viewer.rs` converted from `if/else` to a 3-arm `match`. - `Name` sort uses `nice_name()` — sorts by title first, then category, both ascending. - `SpanEventRef::SelfTime` and `SpanGraphEventRef::SelfTime` (which have no name) sort as `("", "")` under `Name` mode. **WebSocket protocol (backward compatible):** - `-sorted` suffix → `SortMode::Value` (existing clients unaffected) - `-sorted-by-value` suffix → `SortMode::Value` - `-sorted-by-name` suffix → `SortMode::Name` - no suffix → `SortMode::ExecutionOrder` **napi bindings (`next-napi-bindings`):** - `TraceQueryOptions.sort` changed from `Option<bool>` to `Option<String>` (`"value"` | `"name"` | omit). **TypeScript:** - `generated-native.d.ts`: `sort?: 'value' | 'name'` - MCP tool schema: `z.enum(['value', 'name']).optional()` - CLI: `--sort <mode>` with `.choices(['value', 'name'])` replacing the boolean `--sort` flag - Tests updated; added new test cases for `sort: 'name'` and `--sort name` <!-- NEXT_JS_LLM_PR --> Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-17 09:44:12 +02:00
4 people	68bac2a70e	turbopack: reduce EffectStateStorage memory by storing u128 content hashes (#92902) ... ### What? In `turbo-tasks-fs`, the `WriteEffect` and `WriteLinkEffect` structs now store a precomputed `u128` xxh3 content hash instead of returning the full `ReadRef<PersistedFileContent>` / `ReadRef<LinkContent>` from their `Effect::value()` implementations. The associated type `Effect::Value` is changed from `ReadRef<PersistedFileContent>` / `ReadRef<LinkContent>` to `u128` for both effect types. `LinkContent` gains a `DeterministicHash` derive to support hashing via `hash_xxh3_hash128`. ### Why? `EffectStateStorage` stores the last-applied value for each effect key in `last_applied: Mutex<Option<Box<dyn Any + Send + Sync>>>` to enable a double-checked deduplication fast path (skip re-writing a file if content hasn't changed). In long-running sessions this map grows continuously and holds onto a lot of memory because it stores full file content (`ReadRef<PersistedFileContent>`) or full link content (`ReadRef<LinkContent>`) for every file ever written. Storing a `u128` hash (16 bytes) instead of the full content significantly reduces memory usage while preserving the deduplication semantics (hash equality implies content equality with negligible collision probability). ### How? - Added `content_hash: u128` field to `WriteEffect` and `WriteLinkEffect`, computed at construction time using `hash_xxh3_hash128` (xxh3 128-bit non-cryptographic hash, already used elsewhere in turbopack for content addressing). - Changed `Effect::Value = u128` for both effect types; `value()` returns `&self.content_hash`. - The `EffectStateStorage` now stores `Box<u128>` (24 bytes on the heap) per entry instead of `Box<ReadRef<...>>` which held a reference-counted pointer into potentially large file content. - No changes to `EffectStateStorage`, `Effects::apply()`, or `DynEffect` vtable — the existing `Any`/`DynPartialEq` machinery works transparently with `u128`. <!-- NEXT_JS_LLM_PR --> Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-17 09:14:42 +02:00
Tobias KoppersandGitHub	4dd1fcb17a	turbo-tasks: add info_span for blocking waits during task restore (#92878) ... ### What? Adds a `tracing::info_span!("blocking")` around `listener.wait()` calls in `ExecuteContextImpl::task()` when a task is still being restored from backing storage. ### Why? When a task is locked for restoration by another thread, the current thread spins in a loop and calls `listener.wait()` until notified. CPU time accounting treats this waiting time as active CPU usage, which skews profiling results. By wrapping the wait in a `"blocking"` span, the tracing infrastructure can correctly exclude this idle blocking time from CPU time — ensuring CPU metrics reflect actual computation rather than time spent waiting for another thread to finish restoring a task. ### How? Import `tracing::info_span` and enter a `"blocking"` span immediately before `listener.wait()` in the restore-wait loop in `turbopack/crates/turbo-tasks-backend/src/backend/operation/mod.rs`. The span is dropped when the wait returns (or when the loop continues to re-check the condition). <!-- NEXT_JS_LLM_PR -->	2026-04-16 17:43:47 +02:00
5 people	f49722285b	turbopack: cache TransformPlugin in narrow-scoped turbo-tasks functions (#92842) ... ### What? Refactor all usages of `EcmascriptInputTransform::Plugin(ResolvedVc::cell(Box::new(...) as _))` across `next-core` and `turbopack-tests` so that `TransformPlugin` cells are created inside dedicated `#[turbo_tasks::function]` functions rather than inline at call sites. Additionally: - Introduce a `JsonValue` newtype wrapping `serde_json::Value` that implements `TaskInput`, enabling the SWC wasm plugin list to be passed through a turbo-tasks function boundary and properly cached. - Replace the `bool` roundtrip in `next_strip_page_exports` with an `ExportFilterInput` enum that derives `TaskInput`, mirroring `ExportFilter` exhaustively (so a new upstream variant is a compile error, not a silent fallback). - Derive `TaskInput` on `ActionsTransform` and pass it directly to the cached function instead of converting to `is_server: bool` at the call site. - Replace all `.expect("... config must exist")` panics in option-gated plugin functions (`emotion`, `styled_components`, `react_remove_properties`, `remove_console`, `relay`) with `.context(...)?` for proper error propagation. - Add `// TODO: use get_ecma_transform_rule instead` comments to the ~10 transform functions that manually inline the same `ModuleRule::new` + `ExtendEcmascriptTransforms` pattern that `get_ecma_transform_rule` abstracts. ### Why? `TransformPlugin` is not serializable and not comparable. When a `TransformPlugin` is `cell`ed inline (i.e. `ResolvedVc::cell(Box::new(...) as _)`) inside a turbo-tasks function, a new cell is created on every invocation of the enclosing function, because the framework has no way to detect that the value is the same as before. This causes every task that depends on the `TransformPlugin` cell to be invalidated unnecessarily. By moving the `Vc::cell(...)` call into its own narrow-scoped `#[turbo_tasks::function]`, turbo-tasks can cache the cell by the function's inputs. If the inputs haven't changed, the function won't re-run, the existing cell is reused, and downstream tasks are not invalidated. ### How? Each inline `ResolvedVc::cell(Box::new(SomeTransformer { ... }) as _)` is replaced with: ```rust some_transform_plugin(args).to_resolved().await? #[turbo_tasks::function] fn some_transform_plugin(args: ...) -> Vc<TransformPlugin> { Vc::cell(Box::new(SomeTransformer { ... }) as Box<dyn CustomTransformer + Send + Sync>) } ``` Where a cached function needs to store a `ResolvedVc` in the resulting transformer struct, the parameter is declared as `ResolvedVc<T>` directly in the `#[turbo_tasks::function]` signature. The turbo_tasks macro rewrites `ResolvedVc<T>` → `Vc<T>` in the external call-site signature, and the call site passes a dereferenced `*resolved_vc`. This avoids a redundant `.to_resolved().await?` inside the function body. For the SWC wasm plugin case, `serde_json::Value` (used for per-plugin config) doesn't implement `Hash` or `TaskInput`. A `JsonValue` newtype is introduced with: - `#[bincode(with = "turbo_bincode::serde_self_describing")]` for serialization - Manual `Hash` impl that hashes the JSON string representation - Manual `TaskInput` impl (`is_transient = false` since the type contains no `Vc`s) <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Niklas Mischkulnig <[email protected]>	2026-04-16 08:17:22 +02:00
4 people	28f96a4f79	fix(trace-server): default MCP port to --port + 1 to avoid conflicts (#92831) ... ### What? When `--mcp-port` is not explicitly provided to `next internal trace`, the MCP HTTP server now defaults to `--port + 1` (the WebSocket port plus one) instead of the hardcoded value `5748`. The `DEFAULT_WS_PORT` remains `5747`, so when neither flag is passed the behavior is unchanged: WS=5747, MCP=5748. ### Why? Starting the trace server twice with different `--port` values (e.g. to analyze two trace files simultaneously) caused both instances to attempt to bind the same hardcoded MCP port `5748`: ``` Error: MCP port 5748 is already in use. Use --mcp-port to specify a different port. ``` Making the MCP port default relative to the WS port means each invocation with a unique `--port` automatically gets a unique `--mcp-port` without any extra flags. ### How? In `startTurboTraceServerCli`, changed: ```ts const httpPort = mcpPort ?? DEFAULT_MCP_PORT // always 5748 ``` to: ```ts const httpPort = mcpPort ?? wsPort + 1 // e.g. 5748, 6001, 7001, … ``` The now-unused `DEFAULT_MCP_PORT` constant in `turbo-trace-server.ts` was removed. The identical constant in `query-trace.ts` is kept — it is the client-side default for *connecting* to an MCP server and `5748` remains a sensible default there. The `--mcp-port` help text was updated to reflect the dynamic default. <!-- NEXT_JS_LLM_PR --> Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-15 17:41:41 +00:00
Tobias KoppersandGitHub	f8de120d1e	Turbopack: shorter error message for ModuleBatchesGraph::get_entry_index (#92828) ... ### What? In `ModuleBatchesGraph::get_entry_index`, when an entry module is not found in the graph, the error message included the full list of all possible entries (with their resolved string identifiers). This is an expensive operation that requires async resolution of every entry's `ident().to_string()`. ### Why? In production builds, this verbose error output is unnecessary overhead and produces very long error messages that are hard to read. The list of possible entries is only useful during development/debugging. ### How? Gate the expensive "possible entries" diagnostic behind `cfg!(debug_assertions)`. In release builds, a short `bail!("Entry is not in graph")` is used instead. In debug builds, the full diagnostic with all possible entry identifiers is still shown to aid debugging. This mirrors the same approach already applied to `ChunkGroupInfo::get_index_of`. <!-- NEXT_JS_LLM_PR -->	2026-04-15 17:19:46 +02:00
4 people	f760e48532	pr-status: Show all participants in review thread overview (#92730) ... Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-15 10:24:43 +02:00
4 people	b9cd44892c	Add TurboMalloc::thread_park() to flush and collect on thread park (#92804) ... ### What? Adds `TurboMalloc::thread_park()` — a new method on `TurboMalloc` that calls `libmimalloc_sys::mi_collect(false)` to process mimalloc's lazy free lists when a Tokio worker thread parks. Registers it as an `on_thread_park` callback in every Tokio runtime that already registers `on_thread_stop`: - `turbopack-cli/src/main.rs` - `turbopack-cli/benches/small_apps.rs` - `next-napi-bindings/src/lib.rs` - `next-build-test/src/main.rs` - `turbo-tasks-backend/fuzz/src/graph.rs` ### Why? mimalloc uses deferred freeing: memory freed by one thread is not reclaimed immediately but is queued until that thread performs N more allocations. Tokio worker threads that are parked (waiting for work) don't allocate, so deferred frees accumulate indefinitely — memory is never returned to the OS until the thread wakes up and does real work. Calling `mi_collect(false)` on park processes the lazy free lists, allowing mimalloc to reclaim that memory, while avoiding the syscalls and globally synchronized operations that `mi_collect(true)` would trigger. ### How? - `thread_park()` only calls `mi_collect(false)` — it intentionally does **not** call `flush()`. `flush()` transfers the thread-local allocation counter to the global atomic and resets it to zero, which would violate the trace server's assumption that per-thread counters are monotonically increasing. Counter sync remains exclusively in `thread_stop()`. - `thread_park()` is feature-gated behind `cfg(all(feature = "custom_allocator", not(target_family = "wasm")))` and is a no-op otherwise. - `libmimalloc-sys` is added as a direct optional dependency under `[target.'cfg(not(target_family = "wasm"))'.dependencies]` (it was already a transitive dep via `mimalloc`) to access `mi_collect` directly, since `mimalloc`'s `use ffi::*` is not a public re-export. - In each `on_thread_park` callback, `TurboMalloc::thread_park()` is called **after** the existing SWC atom store GC, so atoms freed by that GC pass are also eligible for reclamation in the same collection. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-15 03:43:39 +02:00
4 people	b56f1555d8	turbopack: gate ValueDebugFormat and ValueDebug behind debug_assertions (#92628) ... ### What? Gate `ValueDebugFormat` and `ValueDebug` behind `#[cfg(debug_assertions)]` across the turbopack crates, eliminating all debug formatting machinery from release binaries entirely. ### Why? `ValueDebugFormat` and `ValueDebug` contribute to release binary bloat. The field-level formatting logic (iterating struct fields, resolving `Vc`s, formatting collections recursively) and the per-type `ValueDebug` trait registrations are purely debugging aids with no value in production binaries. **Measured impact: -7.3 MiB (-5.5%) reduction in release `libnext_napi_bindings.so` binary size** (from 126.26 MiB to 119.31 MiB). ### How? **`ValueDebug` trait** (`debug/mod.rs`): - In debug builds: full `#[turbo_tasks::value_trait(no_debug)]` with `dbg()` / `dbg_depth()` methods, `ValueDebugFormatString`, all blanket impls for collections/tuples/etc. - In release builds: empty marker trait with blanket `impl<T: ?Sized> ValueDebug for T {}`. This satisfies the supertrait bound on all value traits at zero cost — no per-type impl code is generated. **`ValueDebugFormat` trait** (`debug/mod.rs`): - The `value_debug_format` method only exists under `#[cfg(debug_assertions)]`. In release builds, the trait is still present (for derive macros to reference) but has no methods. - All blanket impls (`String`, `RcStr`, `Option`, `Vec`, `SmallVec`, `AutoSet`, `AutoMap`, `HashMap`, `FxIndexSet`, `FxIndexMap`, tuples) are gated behind `debug_assertions`. - Supporting infrastructure (`ValueDebugFormatString`, `PassthroughDebug`, `vdbg`, `internal` submodule, `value_debug_format_field`) is compiled away. **Proc-macros**: - `#[derive(ValueDebugFormat)]`: emits a full impl with `value_debug_format` method in debug builds, empty impl in release builds. - `#[derive(ValueDebug)]` and `value_impl` blocks: emit full debug impl in debug builds only — **no release impl at all** (the blanket marker trait impl covers it). - `#[turbo_tasks::value]`: transparent types get `#[cfg(debug_assertions)]` on the manual `ValueDebug` impl. Non-transparent types use `#[cfg_attr(debug_assertions, derive(turbo_tasks::debug::internal::ValueDebug))]` so the `internal` module is never referenced in release. - `#[turbo_tasks::value_trait]`: the `Dynamic/Upcast/UpcastStrict` impls for `Box<dyn ValueDebug>` are gated behind `#[cfg(debug_assertions)]`. **Callers** (`vc/mod.rs`, `vc/resolved.rs`, `read_ref.rs`, `mapped_read_ref.rs`, `macro_helpers.rs`, `alias_map.rs`): - All `impl ValueDebugFormat` blocks and their imports are gated behind `#[cfg(debug_assertions)]`. ### Verification - `cargo check --release` — clean (no errors, no warnings) - `cargo clippy --all-targets` — clean - CI passing <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-15 03:29:20 +02:00
4 people	fff4a4d9bf	turbo-persistence: stop background persisting after unrecoverable failure (#92106) ... ### What? When a persist or compaction operation fails in `turbo-persistence`, the database now: - Rolls back cleanly (deletes orphan files, restores CURRENT) - Stops the background persisting process for the session - Keeps in-memory state consistent with on-disk state at all times - Deletes superseded files safely (with Windows fallback for open memory maps) ### Why? Previously, a failed write operation (e.g. disk full, I/O error) would leave the database in a broken state: 1. **Misleading error loop** — The `active_write_operation` `AtomicBool` was left set to `true` after failure, so every subsequent snapshot cycle printed _"another write operation is already in progress"_ forever, hiding the real error. 2. **In-memory corruption** — `commit()` mutated `inner.meta_files` and `inner.current_sequence_number` *before* writing the CURRENT file to disk. If a disk error occurred between those two steps, the in-memory state was inconsistent with disk and the rollback had no way to fix it. 3. **Rollback could corrupt committed data** — If `commit()` failed *after* writing CURRENT (e.g. during old-file deletion or LOG writing), the rollback would delete the *newly committed* files, corrupting the database. 4. **Task graph corruption** — `save_snapshot` consumes task cache log entries. If it failed, those entries were lost, but the background loop would continue trying to persist — silently skipping those tasks and corrupting the task graph in storage. 5. **Partially written CURRENT** — If the failure happened mid-write to the CURRENT file, it could be left with partial/corrupt content, but nothing restored it. ### How? **`WriteOperationGuard` RAII (db.rs)** A new `WriteOperationGuard<'a>` replaces the `AtomicBool` + manual `try_recover_after_failed_write()` pattern. The guard holds: - `&'a Mutex<Option<ActiveWriteState>>` — the write slot (`None` = idle, `Some(Active("write batch"))` = in progress, `Some(Error)` = permanently disabled) - `path: &'a Path` — database directory for rollback - `seq_before: u32` — sequence number at operation start - `succeeded: bool` — set by `guard.success()` On `drop`, if not succeeded: 1. Writes `seq_before` back to CURRENT (repairs a partially-written CURRENT) 2. Deletes all files with `seq > seq_before` (orphans from the failed operation) 3. Sets the slot to `None` (success) or `Some(Error)` (if cleanup itself failed) The `Active` variant carries a `&'static str` name (e.g. `"write batch"`, `"compaction"`) used in error messages. **Three-phase `commit()` (db.rs)** `commit()` is restructured so `inner` is completely unmodified before the point of no return: | Phase | What happens | `inner` state | On failure | |-------|-------------|---------------|------------| | **A** | Compute `meta_seq_numbers_to_delete` via `sst_filter`. Uses `apply_filter_collect` (read-only) to update filter state and collect per-meta-file removal sets without modifying any MetaFile. Only a read lock on `inner` is needed. | Unchanged | Guard deletes orphan files + restores CURRENT; `inner` is intact | | **B** | Write `.del` file and CURRENT to disk. | Unchanged | Same as above | | **C** | Apply deferred `retain_entries` (from A's removal sets), append new metas, remove obsolete metas, bump `current_sequence_number`. Try to delete superseded files; defer failures. | Updated | CURRENT is already durable; commit is irreversible | After CURRENT is written (point of no return), LOG writing errors are caught and reported via `eprintln!` — they must not propagate because the `WriteOperationGuard` would then run its rollback and delete the *newly committed* files. **`SstFilter::apply_filter_collect` (sst_filter.rs)** A new read-only variant of `apply_filter` that updates the filter state and returns a `FxHashSet<u32>` of SST entry sequence numbers to remove from each meta file, without calling `retain_entries`. The original `apply_filter` (which mutates the MetaFile) is still used by `load_directory` and during new-meta-file construction where immediate mutation is appropriate. **Deferred file deletion (db.rs)** Superseded `.sst`/`.meta`/`.blob` files are deleted immediately after Phase C (once `inner` is updated). On Linux/macOS this always succeeds, even if concurrent readers have the files memory-mapped. On Windows, open memory maps prevent deletion — any file that fails is stored as a `DeferredDeletion` enum (`Sst(u32)` / `Meta(u32)` / `Blob(u32)`) and retried on the next commit or at shutdown. The `.del` file written during Phase B ensures crash recovery via `load_directory` regardless. **Background loop error handling (backend/mod.rs)** - `snapshot_and_persist()` returns `Result<(Instant, bool), anyhow::Error>` instead of `Option`. When `save_snapshot` fails, the error propagates with `?`. - The background loop matches on the `Result`: on `Err`, it logs the error and a message that persisting is disabled for this session, then returns (permanently stopping the background job). - `has_unrecoverable_write_error()` checks the `ActiveWriteState::Error` variant to detect permanent failure after compaction errors. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-14 18:29:40 +02:00
Tobias KoppersandGitHub	54e44b42a9	feat(turbo-tasks-backend): gate prepare_tasks_with_callback trace span behind feature flag (#92779) ... ### What? Gates the `prepare_tasks_with_callback` trace span in `turbo-tasks-backend` behind a new `trace_prepare_tasks` Cargo feature flag. ### Why? The `prepare_tasks_with_callback` span was emitted unconditionally on every call, which is extremely frequent during task scheduling. This made trace files very verbose with high-volume, low-signal entries. All other trace spans in `turbo-tasks-backend` (e.g. `trace_task_completion`, `trace_task_modification`, `trace_task_dirty`, etc.) are already gated behind `#[cfg(feature = "trace_*")]` flags — this brings `prepare_tasks_with_callback` in line with that convention. ### How? - Added `trace_prepare_tasks = []` to the `[features]` section of `turbopack/crates/turbo-tasks-backend/Cargo.toml`, alongside the existing `trace_*` flags. - Gated the `use tracing::trace_span` import and the `trace_span!("prepare_tasks_with_callback", reason)` call in `src/backend/operation/mod.rs` behind `#[cfg(feature = "trace_prepare_tasks")]`. - Added `#[cfg(not(feature = "trace_prepare_tasks"))] let _ = reason;` to suppress the unused-variable warning when the feature is off (the `reason` parameter has no other use in the function body). The span is now silent by default. It can be re-enabled for debugging by compiling with `--features trace_prepare_tasks`. <!-- NEXT_JS_LLM_PR -->	2026-04-14 08:10:25 -07:00
4 people	5ff5ae217b	feat(trace-server): add query CLI and MCP API to turbopack-trace-server (#92030) ... ### What? Adds a [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) HTTP server and a companion CLI to the Turbopack trace server (`next internal trace`), so AI agents and developers can explore trace data from a build or dev session incrementally through a structured tool API. Two new user-facing features: 1. **`--mcp-port` flag on `next internal trace`** — starts an MCP-over-HTTP endpoint (at `/mcp`) alongside the existing WebSocket trace viewer. 2. **`next internal query-trace`** — a standalone CLI command that queries a running trace server from the terminal, using the same MCP endpoint under the hood. ### Why? Trace files from Turbopack builds can be very large. An agent querying raw trace data would be overwhelmed by tens of thousands of spans. The MCP API exposes the data incrementally: - **Pagination** (20 spans per page) prevents context overflow. - **Aggregation** groups repeated spans by name (same logic as the WebSocket viewer's graph), so agents see one entry per span type with count/total/avg statistics rather than thousands of identical entries. - **Drill-down** lets agents pick a span ID from one response and fetch its children in the next call. - **Search** filters spans by name/args using `SpanRef::search` (full subtree search index, same as the WebSocket viewer). - **Sort** orders by corrected (wall-clock) duration to surface the slowest spans first. - **Output format** — `outputType: "json"` for machine-readable structured data, `"markdown"` (default) for human-readable rendering. ### How? #### Rust (`turbopack/crates/turbopack-trace-server/src/lib.rs`) - `start_turbopack_trace_server_non_blocking(path, port) -> Arc<StoreContainer>` — starts the reader and WebSocket server on background threads and returns the store handle immediately (previously the function blocked forever). - `QueryOptions` — struct with `parent`, `aggregated`, `sort`, `search`, `page` fields. - `query_spans(store, opts) -> QueryResult` — queries up to 20 spans per page. In aggregated mode it uses the existing `SpanGraphRef` graph logic (the same grouping the WebSocket viewer uses). Waits up to 10s for the store to finish initial data loading on the first call. - Helper functions extracted: `paginate()`, `format_span_name()`, `build_span_id()` to avoid duplication between aggregated and raw code paths. #### Span ID format IDs encode both the span type and the navigation path: | Span kind | Leaf format | Example full path | |---|---|---| | Raw span | `<index>` (decimal) | `a1-a5-20` | | Aggregated span | `a<first-span-index>` | `a1-a5-a34` | Segments are joined by `-` as the caller drills deeper. `resolve_span_by_id` only needs the last segment to look up the underlying store index. #### NAPI bridge (`crates/next-napi-bindings/src/turbo_trace_server.rs`) - `TraceServerHandle` — opaque NAPI class wrapping `Arc<StoreContainer>`. - `startTurbopackTraceServerHandle(path, port)` — calls the non-blocking Rust function. - `queryTraceSpans(handle, options)` — calls `query_spans` and returns a plain JS object. - Doc comments on `SpanInfo` fields clarify that `cpu_duration`/`corrected_duration` hold the first example span's values for aggregated groups (while `total_*` fields hold group totals). #### TypeScript (`packages/next/src/...`) - `generated-native.d.ts` / `types.ts` / `index.ts` — type declarations, native wrappers, and WASM stubs. - `next.ts` — adds `--mcp-port <port>` to `next internal trace` (defaults to `5748`); registers `next internal query-trace` subcommand with `--json` flag. - `turbo-trace-server.ts` — rewrites the CLI handler to: 1. Start the WebSocket trace viewer server non-blocking (was blocking). 2. Start an HTTP server at `/mcp` running MCP `StreamableHTTPServerTransport` (stateless — one transport per request, `sessionIdGenerator: undefined`). 3. `renderSpanMarkdown()` — extracted helper that renders a single span as markdown. 4. Error handling: `server.on('error')` for `EADDRINUSE`, `try/catch` around `loadBindings()` and `startTurbopackTraceServerHandle()`. - `query-trace.ts` — new CLI command that POSTs JSON-RPC to the MCP endpoint and prints the response. On connection failure, shows instructions to start `next internal trace` first. Supports `--json` flag for structured output. #### `next internal query-trace` CLI ``` Usage: next internal query-trace [options] Options: --port <port> MCP port of the running trace server. Defaults to 5748. --parent <parent> Span ID to enumerate children of. Omit for root level. --no-aggregated Disable aggregation of spans by name (aggregated by default). --sort Sort results by corrected duration descending (default: false). --search <search> Substring filter on span name/category. --json Output as JSON instead of markdown. --page <page> Page number (1-based, default 1). -h, --help Displays this message. ``` #### Startup output When `next internal trace` starts with `--mcp-port`: ``` Turbopack trace server started. View trace at https://trace.nextjs.org?port=5747 Query this trace from the command line: next internal query-trace --help Alternatively, connect an MCP client to http://127.0.0.1:5748/mcp ``` #### Markdown output format (per span) ```markdown ### `<name>` (ID: `<id>`) - CPU Duration: … - Corrected Duration: … - Start (relative to parent): … - End (relative to parent): … **Attributes:** - `key`: value ``` For aggregated spans with count > 1, totals and averages are shown first, then one example span's raw data. ### Tests **E2E test (`test/e2e/turbopack-trace-server-query/turbopack-trace-server.test.ts`)** Uses `nextTestSetup` with `env: { NEXT_TURBOPACK_TRACING: '1' }` to produce a real trace file. Spawns `next internal trace <file> --mcp-port <port>`, waits for the MCP server to be ready, then runs both MCP HTTP and CLI queries: - Root listing (markdown format) - Aggregation mode - Pagination - Drill-down by span ID (using JSON output to extract IDs) - Search with a real span name match and a non-matching term - Sort by duration - JSON output format (`outputType: 'json'`) - CLI: `--sort`, `--search`, `--no-aggregated`, `--parent`, `--json` - Error path: connection failure message with instructions Turbopack-only (skips via `if (!isTurbopack) return` at describe level). The error-path test runs regardless of bundler. ### Usage ```bash # Build with tracing NEXT_TURBOPACK_TRACING=1 next build # Start the trace viewer + MCP server next internal trace .next-profiles/trace-turbopack --mcp-port 5748 # Query from CLI next internal query-trace --sort next internal query-trace --parent a1 --sort next internal query-trace --search "turbo_tasks::function" --page 2 next internal query-trace --no-aggregated next internal query-trace --json # Or connect an MCP client to http://127.0.0.1:5748/mcp ``` <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-14 10:44:07 +02:00
3 people	fa60f50532	turbo-tasks: Fix recomputation loop by allowing cell cleanup on error during recomputation (#92725) ... ### What? Fix an infinite recomputation loop in turbo-tasks by allowing cell cleanup to proceed during recomputation even when the task result is an error. ### Why? When a task errors during normal execution, cell cleanup is intentionally skipped. The rationale is that errors during normal execution are often transient eventual-consistency issues caused by reading stale inputs — keeping old cell data prevents "cell no longer exists" errors for dependent tasks that are reading those cells. However, during **recomputation** (a task re-executing without being marked dirty), nothing about its inputs has changed. Any error it produces is therefore not transient. If cell cleanup is skipped in this case, stale cells remain in place, which trigger further recomputations, which produce further errors, which skip further cleanups — creating an infinite loop. ### How? - Detect whether a task execution is a recomputation by checking `task.is_dirty().is_none()` at completion time (a recomputing task was not dirty when it started executing). - Propagate `is_recomputation` through `TaskExecutionCompletePrepareResult` to both `task_execution_completed_prepare` and `task_execution_completed_cleanup`. - In both places where cell cleanup is gated on `result.is_ok()` / `!is_error`, extend the condition to also allow cleanup when `is_recomputation` is true. This ensures that stale cells produced by a erroring recomputation are cleaned up, breaking the infinite loop. <!-- NEXT_JS_LLM_PR --> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>	2026-04-13 17:52:27 +02:00
5 people	38fdf0c08f	Turbopack: Add import.meta.glob support (Vite compat) (#92640) ... ## What? Adds support for [Vite's `import.meta.glob`](https://vite.dev/guide/features.html#glob-import) in Turbopack. This is a compile-time transform that resolves glob patterns into a map of module paths to lazy/eager imports. ## Why? This is a commonly used Vite feature for dynamically importing groups of files (e.g., all markdown files in a directory, all route modules matching a pattern). ## How? ### Core implementation (`import_meta_glob.rs`) 1. **Analysis phase**: `import.meta.glob` is recognized as a well-known function via `WellKnownFunctionKind::ImportMetaGlob`. When called, arguments are statically analyzed to extract patterns and options. 2. **File discovery**: Uses Turbopack's `read_glob` (with `Glob::can_match_in_directory` for efficient directory pruning) to find matching files. Negative patterns (prefixed with `!`) are applied via a separate `Glob` matcher. 3. **Virtual module**: Each unique `import.meta.glob()` call generates a virtual `ImportMetaGlobAsset` module that exports an object mapping file paths to either: - **Lazy mode** (default): `() => import('./path')` thunks — resolved with `EcmaScriptModulesReferenceSubType::DynamicImport` - **Eager mode**: Direct `require('./path')` results — resolved with `EcmaScriptModulesReferenceSubType::Import` 4. **Code generation**: The `import.meta.glob(...)` call site is replaced with `__turbopack_require__(virtual_module_id)`. ### Architecture - **`ImportMetaGlobAsset`** is the virtual module. It stores only the origin, patterns, and options — no `source` reference. - **`ImportMetaGlobAsset::map()`** is a `#[turbo_tasks::function]` that builds glob matchers, scans the filesystem, and resolves all matched files as ESM imports. Being a turbo-tasks function, the result is memoised. Both `references()` and `chunk_item_content()` call this single cached function. - **`ident()`** is derived from `AssetIdent::from_path(origin_path)` with a modifier encoding all glob options, so two `import.meta.glob()` calls with different options produce different module idents. - **Side effects**: Lazy mode → `SideEffectFree` (only exports thunks, nothing evaluated). Eager mode → `ModuleEvaluationIsSideEffectFree` (the virtual module itself has no side effects, but its synchronous requires trigger real module evaluations). ### Supported options | Option | Description | |--------|-------------| | `eager` | `boolean` — load modules synchronously (default: `false`) | | `import` | `string` — select a named export (e.g., `'default'`) | | `query` | `string` — append query to imports (e.g., `'?raw'`) | | `base` | `string` — base directory for glob scanning | ### Error handling Unsupported or invalid usage produces clear compile-time diagnostics (error code `TP1008`): - `as` option → "not supported, use `query` instead" - Unknown option keys → lists supported options - Non-constant `eager` → "must be a constant boolean" - Non-constant patterns → "must be string literals" ### Not supported (intentionally) - `import.meta.globEager()` (removed in Vite 3) — users should use `{ eager: true }` - `as` option (deprecated in Vite 5) — users should use `query` ## Test Plan - [x] **Turbopack execution test** (`turbopack/crates/turbopack-tests/tests/execution/turbopack/resolving/import-meta-glob/`) - Lazy mode, eager mode, named import (`import: 'default'`), negative patterns - [x] **Turbopack snapshot tests** (`turbopack/crates/turbopack-tests/tests/snapshot/import-meta/glob/`) - Lazy glob, eager glob, named import - Negative patterns (`!**/bar.js`) - Multiple patterns (`['./dir/*.js', './other/*.js']`) - [x] **Turbopack snapshot error tests** (`turbopack/crates/turbopack-tests/tests/snapshot/import-meta/glob-error/`) - `as` option error, non-constant `eager` error, unknown option error - [x] **Next.js e2e tests** (`test/e2e/import-meta-glob/`) - Lazy/eager/named import modules, negative patterns, multi-pattern - Passes in both dev (`next dev`) and production (`next build && next start`) modes - Skipped under webpack (`IS_WEBPACK_TEST=1`) since this is a Turbopack-only feature <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Niklas Mischkulnig <[email protected]>	2026-04-13 17:41:26 +02:00
4 people	143d9ca1ab	feat(turbopack): change DiskFileSystem root to Vc<RcStr> for portable cache (#92603) ... ### What? Changes the `root` parameter of `DiskFileSystem::new`, `DiskFileSystem::new_with_denied_paths`, and the internal `DiskFileSystem::new_internal` from `RcStr` to `Vc<RcStr>`. ### Why? Part of the effort to make the Turbopack persistent cache portable when the absolute path of the project directory changes. By accepting the root as a `Vc<RcStr>` instead of a plain `RcStr`, the path is tracked as a turbo-tasks value. This allows the cache to remain valid even when the project root moves (e.g. on a different machine, a different CI working directory, or a relocated checkout), since the value can be substituted independently without invalidating unrelated cached computations. ### How? - `new_internal` is now an `async fn` that awaits the `Vc<RcStr>` to extract the underlying `RcStr` for constructing `DiskFileSystemInner`. It returns `Result<Vc<Self>>` accordingly. - `new` and `new_with_denied_paths` (the public constructors, which are not `#[turbo_tasks::function]` themselves) are updated to accept `Vc<RcStr>` and forward it to `new_internal`. - All 23 call sites across the codebase are updated to wrap their root string with `Vc::cell(...)` before passing it. Verified with `cargo clippy --all-targets` (clean), `cargo test -p turbo-tasks-backend` (all pass), and `cargo test -p turbopack-tests` (all 89 pass). <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-11 07:55:58 +02:00
4 people	74cdcde87e	feat(turbopack): add LocalPathOrProjectPath PostCSS config resolution (#91338) ... ### What? Adds a new `experimental.turbopackLocalPostcssConfig` option and corresponding `PostCssConfigLocation::LocalPathOrProjectPath` variant so Turbopack can resolve `postcss.config.js` starting from the CSS file's directory first, falling back to the project root. Also fixes an issue that surfaces when multiple per-directory `postcss.config.js` files are used: **Evaluate pool asset name collisions** — output file names for the Node.js evaluate pool are now derived from an xxh3 hash of the full module ident rather than the bare file name, so multiple `postcss.config.js` files in different directories no longer collide. ### Why? Currently Turbopack uses `ProjectPathOrLocalPath` which checks the project root first, then falls back to the local directory. This means per-directory `postcss.config.js` files cannot override the root config — the root config always wins. For projects that need different PostCSS transforms in different directories (e.g. a monorepo with multiple apps or style directories), the more specific config should take precedence. ### How? **Turbopack PostCSS resolution** (`turbopack/crates/turbopack-node/src/transforms/postcss.rs`): - Added `PostCssConfigLocation::LocalPathOrProjectPath` enum variant with doc comments on all variants - Refactored `find_config_in_location()` to express the search order as a `Vec<FileSystemPath>` built from a single `match`, making each variant's strategy immediately clear **Evaluate pool asset file naming** (`turbopack/crates/turbopack-node/src/evaluate.rs`): - Replaced file-name-based output path derivation with a deterministic xxh3 hash of the full module ident - Adds `turbo-tasks-hash` dependency to `turbopack-node` - Eliminates name collisions when multiple entries share the same file name (e.g. per-directory `postcss.config.js` files) **Next.js config flag** (`packages/next/src/server/config-shared.ts`, `config-schema.ts`): - Added `experimental.turbopackLocalPostcssConfig: boolean` option (opt-in, default `false`) - Wired through `crates/next-core/src/next_config.rs` → both client and server contexts (`next_client/context.rs`, `next_server/context.rs`) - When `true`, uses `LocalPathOrProjectPath`; when `false`/unset, uses the existing `ProjectPathOrLocalPath` (no behavior change by default) **E2E test** (`test/e2e/app-dir/turbopack-postcss-multiple-configs/`): - Runs in both **Turbopack dev and production modes** — skips webpack only (webpack does not support function-valued PostCSS plugins and the `turbopackLocalPostcssConfig` flag is Turbopack-specific) - 5 style directories, each with its own `postcss.config.js` passing a **unique color option** (`blue`, `purple`, `orange`, `cyan`, `magenta`) to a shared PostCSS plugin, proving that per-directory config resolution produces distinct results rather than all sharing the same transform - Root `postcss.config.js` is a no-op — if only the root config were used, CSS would remain unmodified - Test validates all 15 elements (5 dirs × 3 files) render with CSS module classes - Test collects all CSS (inline `<style>` tags for dev, linked `.css` files for production) and asserts each directory's expected color is present and `color: red` is absent **Documentation** (`docs/01-app/03-api-reference/05-config/01-next-config-js/turbopackLocalPostcssConfig.mdx`, `docs/01-app/03-api-reference/08-turbopack.mdx`): - New dedicated reference page for `experimental.turbopackLocalPostcssConfig` covering usage, behavior table, and a directory-tree example - Added to the experimental options table in the Turbopack API reference ## Test plan - [x] `cargo check -p turbo-tasks-backend` passes - [x] `cargo check -p turbopack-node` and `cargo check -p next-core` pass - [x] E2E test `turbopack-postcss-multiple-configs` validates per-directory PostCSS config resolution across 5 directories with distinct per-directory colors (Turbopack dev and production modes) - [x] Test correctly skips in webpack mode and deployment - [x] Documentation page and experimental options table updated <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-04-11 07:55:10 +02:00
4 people	4b3e4e8adc	turbopack: move CSS module composes validation from code generation to resolving (#92582) ... ### What? Move the validation that can produce `CssModuleComposesIssue` from code generation (`chunk_item_content`) to the reference's own `resolve_reference()` method, so these errors are surfaced during resolving rather than only during bundling. Two validation checks are moved into `CssModuleComposeReference::resolve_reference()`: 1. A `composes: ... from "...";` target module can't be resolved (unresolvable reference) 2. A `composes: ... from "...";` target module is not a CSS module (e.g. composing from a `.js` or `.txt` file) The `IssueStage` is updated from `CodeGen` to `Resolve` to reflect the new phase. Code generation still handles the error cases gracefully (skipping broken references with `continue`) — it just no longer re-emits the issue since it is already emitted by `resolve_reference()`. ### Why? `get_compilation_issues` (used by the MCP server and other tooling) only builds the module graph — it does not run code generation. Previously, both `CssModuleComposesIssue` variants were only emitted during `chunk_item_content()`, which meant they were invisible to `get_compilation_issues`. Developers using the MCP `get_compilation_issues` tool would not see errors from broken `composes` references until an actual build/bundle was triggered. Since `resolve_reference()` is called as part of module graph traversal, emitting the issue there means it is captured by `get_compilation_issues` alongside other resolve-phase errors. ### How? **`turbopack-css/src/references/compose.rs`** - `CssModuleComposeReference::resolve_reference()` is changed from `fn` to `async fn`. It calls `css_resolve` as before, then awaits `first_module()` on the result and validates: - Resolved to nothing → emit "can't be resolved" issue - Resolved to a module that doesn't implement `CssModuleComposable` → emit "not a CSS module" issue - `CssModuleComposesIssue` and its `Issue` impl are moved here from `module_asset.rs`, now using `ResolvedVc<FileSystemPath>` (from `origin_path()`) as the issue location rather than the `IssueSource` of the composing file. - A new `CssModuleComposable` marker trait (`#[turbo_tasks::value_trait]`) is defined here. `resolve_reference()` uses a `try_sidecast` to this trait to check whether the resolved module is a valid compose target — avoiding a hard dependency on `EcmascriptCssModule` from within `compose.rs`. **`turbopack-css/src/module_asset.rs`** - `EcmascriptCssModule` implements `CssModuleComposable`, marking it as a valid `composes:` target. - `module_references()` is reverted to its original simple form — it only collects references without any validation. - All issue-related imports (`Issue`, `IssueExt`, `IssueSource`, etc.) are removed. **Test fixtures** (`test/development/mcp-server/fixtures/compilation-errors-app/app/css-composes-error/`) - `styles.module.css` — CSS module with `composes: something from './not-a-css-module.txt'` - `not-a-css-module.txt` — a plain text file (resolves but is not a CSS module) - `page.tsx` — page importing the broken CSS module **`test/development/mcp-server/mcp-server-get-compilation-issues.test.ts`** - New test case `should detect CSS module composes errors` verifies the issue is surfaced via `get_compilation_issues`. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-10 21:11:29 +02:00
4 people	a98213cbfa	Refactor effects system: dedup/conflict detection, simplified Effect trait (#92300) ... ### What? Refactors the turbo-tasks effects system to support duplicate detection, conflict detection, and state tracking across effect applications. Also adds a ptr_eq fast path to `ReadRef::PartialEq`. ### Why? The existing effects system had several problems: - **No duplicate detection**: The same effect (e.g., writing the same content to the same path) would be re-applied every time, wasting I/O - **No conflict detection**: Two effects writing different content to the same path could cause an invalidation loop where the two writes invalidate each other, keeping a CPU busy-looping. On production builds this would crash with `Dependency tracking is disabled so invalidation is not allowed` - **Unnecessarily generic**: The old `DynEffect`→`DynEffectApplyFuture` chain wrapped errors in `Arc<dyn EffectError>`, plus a `DynEffectApplyFuture` chain, all to work around the fact that `anyhow::Error` doesn't implement `std::error::Error` - **Complex dedup state machine**: `EffectInstance` used a `Mutex<EffectState>` with 4 states (`NotStarted`/`Started`/`Finished`/`Invalid`), `Event`/`EventListener`, and `spawn()` to ensure idempotent application — all of which is now unnecessary since dedup is handled at the `Effects::apply` level - **Task-local context**: `ApplyEffectsContext` was a task-local typed map used only for a directory-creation cache, requiring `spawn()` + scope propagation - **Slow `ReadRef` equality**: `ReadRef::PartialEq` always performed deep content comparison even when both refs pointed to the same allocation ### How? #### Changed files | File | Change | |------|--------| | `turbo-tasks/src/effect.rs` | Rewrote `Effect` trait, `DynEffect`, `EffectStateStorage`, `Effects::apply()` | | `turbo-tasks/src/lib.rs` | Updated re-exports (`EffectError`, `EffectStateStorage`) | | `turbo-tasks/src/read_ref.rs` | Added `ptr_eq` fast path to `ReadRef::PartialEq` | | `turbo-tasks-fs/src/lib.rs` | Updated `WriteEffect`/`WriteLinkEffect` impls, restored `AnyhowWrapper`, removed `register_write_invalidator` | | `turbo-tasks-fs/src/invalidator_map.rs` | Simplified — removed `WriteContent` enum, invalidators now use `()` values | | `turbo-tasks-fs/src/invalidation.rs` | Deleted — `Write`/`WriteKind` invalidation reason types no longer needed | #### New `Effect` trait (in `turbo-tasks/src/effect.rs`) ```rust pub trait Effect: TraceRawVcs + NonLocalValue + Send + Sync + 'static { type Error: EffectError; type Value: Clone + DynPartialEq + Eq + Send + Sync + 'static; fn key(&self) -> Vec<u8>; fn value(&self) -> &Self::Value; fn state_storage(&self) -> &EffectStateStorage; fn apply(&self) -> impl Future<Output = Result<(), Self::Error>> + Send; } ``` The `Error` associated type uses `dyn std::error::Error` (via `EffectError`) rather than `anyhow::Error` to encourage structured error types that can be downcast into `Issue`s — particularly for filesystem errors from `turbo-tasks-fs`. The current fs implementations use `AnyhowWrapper` as a bridge. #### `EffectStateStorage` New struct stored on `DiskFileSystemInner`, using a two-level locking scheme for performance: - `DashMap<Vec<u8>, Arc<EffectStateEntry>>` tracks per-key state - Each `EffectStateEntry` has a sync `parking_lot::Mutex<Option<Box<dyn Any>>>` for the last applied value (fast dedup reads) and a `tokio::sync::Mutex<()>` write lock (serializes concurrent writes to the same key) - The sync fast path avoids `.await` entirely when the stored value matches, critical for high-iteration scenarios (e.g., the `writeToDisk` test that calls `Effects::apply` 10,000+ times per route) #### `Effects::apply()` flow 1. Group effects by `key()`, cache the `(index, Arc<EffectStateEntry>)` pairs in a `OnceLock` — computed once on first call, reused on subsequent calls with **no DashMap lookup** in the hot path 2. Detect duplicates (same key + same value → keep one) and conflicts (same key + different value → error) 3. For each unique effect: - **Sync fast path**: check `last_applied` via `parking_lot::Mutex` — return immediately if value unchanged - **Slow path**: acquire `tokio::sync::Mutex` write lock, re-check, clear `last_applied` to `None` (prevents stale fast-path matches during write), apply the effect, then store the new value #### Equality dispatch `eq_value_dyn` now calls `turbo_dyn_eq_hash::DynPartialEq::dyn_partial_eq` instead of a manual downcast, using the existing `turbo-dyn-eq-hash` crate machinery. #### `ReadRef::PartialEq` ptr_eq fast path ```rust fn eq(&self, other: &Self) -> bool { Self::ptr_eq(self, other) || Self::as_raw_ref(self).eq(Self::as_raw_ref(other)) } ``` Short-circuits deep comparison when both `ReadRef`s share the same `Arc` allocation — the common case in the effects dedup path where the stored and new `ReadRef` originate from the same turbo-tasks cell. #### `Effect::value()` returns `&Self::Value` Avoids cloning `ReadRef` (Arc increment) on every `eq_value_dyn` comparison. The reference goes straight to the stored field. Only `value_dyn()` (called when actually storing a new value after a write) clones. #### Simplified internals - `EffectInstance` reduced from `Mutex<EffectState>` state machine to plain `Box<dyn DynEffect>` - `DynEffect` trait gains `key()`, `eq_value_dyn()`, `value_dyn()`, `state_storage()` for object-safe dispatch; `dyn_apply()` converts `Self::Error` to `anyhow::Error` via the `Into` blanket impl - `Effects::apply()` now performs grouping, dedup, conflict detection, and per-key state-aware application #### Removed - `EffectState` enum (4 variants) + associated event machinery - `ApplyEffectsContext` + `APPLY_EFFECTS_CONTEXT` task-local - `DiskFileSystemApplyContext` struct - `register_write_invalidator` (complex write-invalidation tracking) - `WriteContent` enum from `InvalidatorMap` - `Write`/`WriteKind` invalidation reason types (deleted `invalidation.rs`) #### Kept from original design - `EffectError` trait + `AnyhowWrapper` in `turbo-tasks-fs` — preserved for future structured filesystem error types (to produce typed `Issue`s from write failures) - `invalidate_from_write` — simplified to just remove+fire path invalidators for read-task invalidation after writes - Directory creation cache — moved from task-local `ApplyEffectsContext` to `DashMap<PathBuf, ()>` on `DiskFileSystemInner` #### Test changes - `test_symlink_stress`: Updated to deduplicate random updates per symlink index before writing (the test previously relied on silent conflict resolution) - `is_sync_and_send` test: Relaxed to `is_send` since the new `Effects::apply()` future holds a `Pin<Box<dyn Future + Send>>` across await points (not `Sync`, but `Send` is sufficient) ### Testing - [x] `cargo check --workspace` — zero errors, zero warnings - [x] `cargo clippy -p turbo-tasks --all-targets -- -D warnings -A deprecated` — clean - [x] `ast-grep scan` — clean (all `Err(anyhow!())` replaced with `bail!()`) - [x] `cargo test -p turbo-tasks --lib` — 50 passed - [x] `cargo test -p turbo-tasks-fs --lib` — 110 passed - [x] `cargo test -p turbo-tasks-backend --lib` — 35 passed - [x] `cargo test -p turbopack-tests` — 89 passed <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-10 13:52:39 +02:00
4 people	69264a763f	test: reduce writeToDisk memory test iterations to fix CI timeout (#92586) ... ### What? Reduces `RUNS` from 10,000 to 1,000 in the `next.rs api writeToDisk multiple times` test in `test/development/basic/next-rs-api.test.ts`. ### Why? The test was frequently timing out on CI. It spawns a child `node --expose-gc` process that calls `writeToDisk()` in a loop (`RUNS` times per batch) for each discovered route (~11 routes), with up to 11 measurement batches per route. At 10,000 runs per batch that is up to ~1.1 million `writeToDisk()` calls total — all under the global 60-second Jest timeout. Each call, even in the turbo-tasks memoized steady state, has non-trivial overhead: an NAPI crossing, a task-cache lookup, a `read_strongly_consistent()` wait, and an O(N effects) iteration over output assets (each requiring a mutex acquisition). On slow CI hardware this easily exceeds 60 seconds. ### How? Reduce `RUNS` to 1,000. This gives a 10× speedup without compromising the test's ability to detect memory leaks: a leak of even a single OS page (4 KB) per 1,000 calls is still detectable via the RSS delta check. The warmup + measurement loop structure is unchanged. <!-- NEXT_JS_LLM_PR --> Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-10 08:52:46 +00:00
4 people	7d09d1e605	Fix NextFontIssue IssueStage: CodeGen → Resolve (#92588) ... ### What? Changes `NextFontIssue`'s `IssueStage` from `CodeGen` to `Resolve`. ### Why? `NextFontIssue` is emitted during font resolution — when validating font configuration options (`util.rs`) and when fetching/resolving Google Font metadata (`google/mod.rs`, `google/font_fallback.rs`). None of these occur during code generation; they all happen at the resolve stage of the build pipeline. Using `IssueStage::CodeGen` was incorrect and caused these font issues to be grouped/sorted with code generation errors rather than resolution errors. The local-font counterpart (`FontResolvingIssue` in `next_font/local/mod.rs`) already correctly uses `IssueStage::Resolve`. ### How? Single-line fix: change `IssueStage::CodeGen.cell()` to `IssueStage::Resolve.cell()` in the `stage()` method of `impl Issue for NextFontIssue`. <!-- NEXT_JS_LLM_PR --> Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-10 10:47:04 +02:00
Tobias KoppersandGitHub	1df2586296	turbopack: fix incremental build when module factories are restored from persistent cache (#92569) ... ### What? Fix an incremental build bug in Turbopack where module factories restored from the persistent cache would skip a necessary code conversion step, producing incorrect output. ### Why? `cell_persisted()` previously returned `Vc<Code>` by immediately calling `to_code()` on the newly created `PersistedCode` cell. This meant the returned `Vc<Code>` was a local, non-persisted cell — so it triggered a recomputation of the task that produces the `Vc<Code>`, which recomputes the actual code and doesn't restore it from the PersistentCode. ### How? - Changed `cell_persisted()` to return `ResolvedVc<PersistedCode>` instead of `Vc<Code>`, so callers explicitly hold a handle to the persisted cell rather than a pre-converted `Code`. - Made `PersistedCode::to_code()` `pub` so callers can invoke the conversion when they actually need a `Vc<Code>`. - Updated `module_factory()` and `module_factory_with_code_generation_issue()` to return `ResolvedVc<PersistedCode>` all the way up the call chain, deferring the `to_code()` call to the final `code()` method that produces a `Vc<Code>` for the chunk pipeline. This ensures that on a cache-warm incremental build, turbo-tasks sees the `PersistedCode` task as a dependency and correctly re-runs the `to_code()` conversion when needed, rather than serving a stale or missing `Code` value. <!-- NEXT_JS_LLM_PR -->	2026-04-09 19:52:08 +02:00
4 people	2800d2a7af	turbo-tasks-backend: prevent duplicate task restores with restoring bits (#92389) ... ### What? Adds `data_restoring` and `meta_restoring` transient flag bits to `TaskStorage`, a shared `restored: Event` on `Storage`, and a synchronous `EventListener::wait()` primitive. These are used to coordinate concurrent restore attempts so the same task is never loaded from the backing store more than once simultaneously. ### Why? When multiple threads race to access an unrestored task, each thread would independently call into the backing storage to load the task's data. This causes redundant I/O and can result in one thread's write being silently discarded when another thread also applies a restore for the same task. The fix ensures only one thread performs the actual I/O while others wait for it to finish. ### How? **New flag bits** (`storage_schema.rs`): `meta_restoring` and `data_restoring` are transient (not persisted) bits that track whether a restore is currently in progress for each category of a task. `meta_restored` and `data_restored` are also transient bits that indicate when a category has been fully restored. **New `restored` event** (`storage.rs`): A single `Event` on `Storage` that is notified after every restore attempt (success or failure). Waiting threads subscribe to it, then re-check the flags under lock after waking. **`EventListener::wait()`** (`event.rs`): A synchronous blocking wrapper around `event_listener`'s `.wait()`, needed because restore logic runs in synchronous backend operation contexts (not async). Includes both non-`hanging_detection` and `hanging_detection` builds. **Restore protocol** (`operation/mod.rs`): All three restore entry points (`task()`, `task_pair()`, `prepare_tasks_with_callback()`) follow the same pattern: 1. **Classify under lock** — for each category that needs restoring: if the `*_restoring` bit is already set, mark the task as "wait"; otherwise, set the bit and claim the I/O. 2. **Drop lock and do I/O** — only the thread that claimed a category performs the backing-store lookup. 3. **Wait for other threads** — done *after* our own I/O so both threads' work can overlap. Uses a fast path (check flags under lock before registering a listener) to skip listener allocation when the other thread has already finished. Returns the `StorageWriteGuard` directly to the caller to avoid a second lock acquisition. 4. **Apply results under lock** — merge the loaded data into the task via `apply_restore_result()` (returns `Result<()>`), set `*_restored`, clear `*_restoring`. 5. **Notify once after all tasks** — a single `notify(usize::MAX)` on the `restored` event after all I/O results have been applied, so waiters are unblocked as late as possible but all at once. **Batch restores** (`prepare_tasks_with_callback`): Uses a `TaskRestoreEntry` struct to track per-task state through phases: - **Phase 1a** — classify each task under lock: already restored → callback immediately; another thread restoring → mark as wait; unclaimed → set `*_restoring` bit and claim. - **Phase 1b** — batch I/O for all claimed tasks (single-task or batch backing-store API). - **Phase 1c** — apply I/O results under lock, clear restoring bits. Errors are collected (rather than panicking immediately) so all restoring bits are cleared first—otherwise other threads waiting on those bits would hang. - **Phase 2** — callbacks for self-restored tasks (separated from 1c so the notify happens first and other threads are unblocked as early as possible). Also inserts task types into the task cache. - **Phase 3** — per-task loop: for each task being restored by another thread, wait (using `wait_for_restore_or_panic`), then immediately call the callback with the returned write guard. A fast path at the top of `prepare_tasks_with_callback` short-circuits the entire pipeline when `!should_restore()`, marking tasks as restored and calling callbacks directly. **Shared helpers:** - `apply_restore_result()` — returns `Result<()>`; clears the restoring bit and merges storage on success, or returns `Err` for the caller to handle after notifying waiters. - `wait_for_restoring_task()` — fast path (check under write lock, return guard if already restored) + slow path (listener loop). Accepts `TaskDataCategory` to handle waiting for both categories in a single call with `All`. - `wait_for_restore_or_panic()` — wrapper that panics on error, returning the `StorageWriteGuard` on success. **Other changes:** - `TurboPersistence` (`turbo-persistence/src/db.rs`): removed the `is_empty: AtomicBool` field; emptiness is now checked via a read lock on inner state, avoiding a stale flag that could prevent restores after writes. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-09 09:31:14 +02:00
4 people	1297f606be	perf: use whole_app_module_graph for get_compilation_issues MCP tool (#92473) ... ### What? Optimizes the `get_compilation_issues` MCP tool to use a single whole-app module graph computation instead of iterating all endpoint groups and computing per-route module graphs individually. ### Why? The `get_compilation_issues` MCP tool was too slow for large apps because `get_all_compilation_issues_inner_operation` called `endpoint_group.module_graphs()` for every endpoint group, computing separate per-route module graphs. This is O(routes) graph computations when a single whole-app graph would suffice. The reason the original code didn't use `whole_app_module_graphs()` was that it calls `drop_issues()` in development mode (to prevent every route from re-reporting all issues during HMR). This would suppress the issues the MCP tool needs to collect. ### How? - **New method `Project::whole_app_module_graphs_without_dropping_issues()`** — calls the same underlying `whole_app_module_graph_operation` (the `#[turbo_tasks::function(operation, root)]` computation) but connects the result without dropping issues. Issues propagate normally so callers can collect them. - **`get_all_compilation_issues_inner_operation`** now calls this new method instead of iterating endpoint groups, reducing the work to a single graph computation. - **Extracted `scale_down_node_pool` helper** — the Node.js process pool scale-down logic (shared between both `whole_app_module_graphs` variants) is extracted into a free function to avoid duplication. - **Existing `whole_app_module_graphs()`** is unchanged in behavior — still drops issues in dev mode for HMR per-route subscriptions. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-08 16:32:56 +02:00
4 people	6d572c27e0	scripts/pr-status: add reply-and-resolve-thread command (#92450) ... ### What? Adds a `reply-and-resolve-thread` subcommand to `scripts/pr-status.js` that combines replying and resolving a review thread in a single command. Also updates the generated `thread-N.md` files to show this command alongside the existing separate `reply-thread` and `resolve-thread` options, and documents the command in the agent skill files. ### Why? When addressing PR review comments, the typical workflow is to reply with a description of what was done and then immediately resolve the thread. This previously required two separate commands: ```bash node scripts/pr-status.js reply-thread <id> "Done -- ..." node scripts/pr-status.js resolve-thread <id> ``` Having a single command reduces friction for agents (and humans) closing out review threads after addressing feedback. ### How? - Added `reply-and-resolve-thread` subcommand in `main()` that calls the existing `replyToThread()` and `resolveThread()` functions in sequence. - Updated `generateThreadMd()` to include a "Reply and resolve in one step" code block in the Commands section of each unresolved thread file (alongside the existing separate commands, not replacing them). - Updated `.agents/skills/pr-status-triage/SKILL.md` to mention the combined command in workflow step 6 and added a "Thread interaction" quick-command reference section. - Updated `.agents/skills/pr-status-triage/workflow.md` to show the one-step alternative in the "Resolving Review Threads" section. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-07 20:36:15 +02:00
4 people	8db9a75243	turbo-tasks: fix hashed cell mode crash on task error (re-land #91576) (#92108) ... ### What? Re-lands #91576 ("turbo-tasks: add hashed cell mode for hash-based change detection without cell data"), which was reverted in #92103 due to a `FATAL` crash in the `filesystem-cache` test suite. Includes a bug fix on top: in `task_execution_completed_prepare`, skip updating `cell_type_max_index` when the task completed with an error. Also adds a `CellHash = [u8; 16]` type alias (requested in review) used throughout the hash pipeline. ### Why? **The original feature** (`serialization = "hash"` on `FileContent` and `Code`) stores a hash of the cell data instead of the full serialized value. On session restore, the hash is used to detect whether cell content has changed without needing the full data in memory. This avoids a large persistent cache size increase. **The bug** that caused the revert: When a task fails partway through re-execution (before recreating all the cells from its previous run), `cell_counters` only reflects the partially-executed state. The old code used those partial counters to update `cell_type_max_index`, removing entries for cell types that were not yet created at the point of failure. This caused downstream tasks that still held cell dependencies from the previous successful run to hit a hard "Cell no longer exists" error. **Concrete failure path** in `filesystem-cache rename app page` test: 1. `get_app_page_entry` runs for `/remove-me/page`, creating two `FileContent` cells (indices 0 and 1). `cell_type_max_index[FileContent] = 2` is persisted. 2. The folder is renamed (`app/remove-me` → `app/add-me`), dirtying the task. 3. On re-execution, `get_app_page_entry` fails at `config.await?` (the loader tree errors because the directory is gone) — before any `FileContent::cell()` calls. 4. `cell_counters` has no `FileContent` entry → old code removed `cell_type_max_index[FileContent]`. 5. The `parse` task tries to read `FileContent` cell 1 from `get_app_page_entry` → `cell_type_max_index` is `None` → **"Cell no longer exists" panic → FATAL error**. **Why it didn't crash before** `serialization = "hash"`: `FileContent` was previously serializable, so `parse` read stale cell data directly from `persistent_cell_data`, which `task_execution_completed_cleanup` already preserves on error. With `serialization = "hash"`, data is transient — readers fall back to `cell_type_max_index` for range validation, where a stale `None` caused the crash. ### How? #### Core feature: `serialization = "hash"` cell mode - New `SerializationMode::Hash` variant in `turbo-tasks-macros` — marks a value type as non-serializable but stores a `DeterministicHash` of the cell data for change detection. - `VcCellHashedCompareMode<T>` cell mode: compares values via `PartialEq` when available, falls back to hash comparison when transient data has been evicted. - `hashed_compare_and_update` / `hashed_compare_and_update_with_shared_reference` on `CurrentCellRef` compute and pass content hashes through the update pipeline. - Backend `update_cell` uses hash-based comparison to skip invalidation when the old cell data is unavailable but the hash matches. - `cell_data_hash: AutoMap<CellId, CellHash>` field in task storage persists hashes across sessions. - Stale `cell_data_hash` entries are cleaned up in `task_execution_completed_cleanup` alongside cell data removal. - `CellHash = [u8; 16]` type alias keeps alignment at 1 byte to avoid padding growth in `AutoMap`/`LazyField` enum variants. - Hash bytes use little-endian encoding (`to_le_bytes`) for cross-platform cache portability. #### Bug fix: preserve `cell_type_max_index` on task error In `task_execution_completed_prepare`, guard the `cell_type_max_index` update block with `if result.is_ok()`. This mirrors the existing `task_execution_completed_cleanup` behavior that already skips cell data removal when `is_error` is true, keeping `cell_type_max_index` consistent with the preserved transient cell data. #### Applied to `FileContent` and `Code` - `FileContent` uses `serialization = "hash"` — full content is persisted via a separate `PersistedFileContent` type when needed (e.g., in `DiskFileSystem::write`). - `Code` uses `serialization = "hash"` with `Arc<Vec<Mapping>>` for cheap cloning. `Code::cell_persisted()` creates a `PersistedCode` cell directly and returns `Vc<Code>` via `PersistedCode::to_code()`, avoiding an intermediate hash-mode cell. #### Other improvements - `DeterministicHash` impls for `SmallVec` and `()`. - `Xxh3Hash128Hasher::finish_bytes()` method returning `[u8; 16]`. - `hash = "manual"` option on `#[turbo_tasks::value]` to opt out of auto-deriving `DeterministicHash`. **Note:** The shutdown hang and cache poisoning fixes that were previously on this branch have been merged separately via #92254. ### Test plan - [x] `test/e2e/filesystem-cache/filesystem-cache.test.ts` passes (all 17 tests) - [x] New `turbopack/crates/turbo-tasks-backend/tests/hashed_cell_mode.rs` integration test verifies hash-based change detection: value changes trigger invalidation, equal values (same hash) do not - [x] `cargo check` passes for `turbo-tasks`, `turbo-tasks-backend`, `turbo-tasks-fs`, `turbopack-core`, `turbopack-ecmascript` - [x] CI green (attempt 2) <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-07 17:59:05 +02:00
4 people	985394417f	Fix styled-jsx race condition: styles lost due to concurrent rendering (#92459) ... ### What? Fix a race condition in the Pages Router SSR path where styled-jsx styles were dropped from the rendered HTML. ### Why? `styledJsxInsertedHTML()` reads and flushes the styled-jsx style registry. Previously it was called concurrently with the page render via `Promise.all`: ```js const [rawStyledJsxInsertedHTML, content] = await Promise.all([ renderToString(styledJsxInsertedHTML()), (async () => { /* render the page */ })(), ]) ``` Because both ran at the same time, `styledJsxInsertedHTML()` could (and in practice did) execute and flush the registry **before** the page render had finished populating it. The result was that dynamic styled-jsx styles — those with interpolated expressions that compute their class names at runtime via DJB2 hashing — were silently dropped from the SSR output, causing a flash of unstyled content on first load. This is particularly visible in production deployments where all components use dynamic styled-jsx (numeric `jsx-*` class names), since those styles only exist in the registry after rendering completes. ### How? Serialize the two operations: render the page first, then call `styledJsxInsertedHTML()`. Since the registry is fully populated by the time it is read, all styles are captured correctly. A new e2e test (`test/e2e/styled-jsx-dynamic`) exercises this scenario with multiple nested components that all use dynamic styled-jsx with interpolated props, covering the exact FOUC pattern seen in production. <!-- NEXT_JS_LLM_PR --> Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-07 17:02:37 +02:00
4 people	287fd7e704	turbo-persistence: pin referenced BlockCache entries to prevent eviction (#92361) ... ### What? Implements [`quick_cache::Lifecycle`](https://docs.rs/quick_cache/latest/quick_cache/#lifecycle-hooks) hooks for the `BlockCache` in turbo-persistence. A new `BlockCacheLifecycle` struct overrides `is_pinned` to check whether the `ArcBytes` value's backing `Arc` has a strong count > 1, indicating it is still referenced outside the cache. When `is_pinned` returns `true`, the cache's eviction algorithm (Clock-PRO / S3-FIFO) skips the entry instead of evicting it. ### Why? Without pinning, the cache can evict blocks that are still in use — the caller holds an `ArcBytes` clone (which shares the `Arc` backing), so the data isn't lost, but the cache slot is wasted. When the caller finishes and the entry is looked up again, it results in an unnecessary cache miss and re-read/decompress from disk. By checking `Arc::strong_count > 1` in `is_pinned`, we avoid evicting entries that are actively referenced, improving cache hit rates under concurrent access. ### How? **`ArcBytes::is_shared()`** (`arc_bytes.rs`): - Returns `true` when the backing `Arc<[u8]>` has `strong_count > 1` (i.e., someone outside the cache holds a clone). - Mmap-backed entries always return `false` — the mmap `Arc` is shared across all slices from the same file, so checking its `strong_count` would make them permanently unevictable. This is fine because #92390 already drops mmap blocks from the cache entirely. **`BlockCacheLifecycle`** (`static_sorted_file.rs`): - Implements `quick_cache::Lifecycle<(u32, u16), ArcBytes>` with `RequestState = ()`. - `is_pinned` delegates to `val.is_shared()`. - `begin_request` and `on_evict` are no-ops since we only need the pinning behavior. **`BlockCache` type alias** updated to include `BlockCacheLifecycle` as the 5th generic parameter. All call sites use `Default::default()` for construction, so no call-site changes were needed. **Test cleanup**: Removed the redundant `TestBlockCache` type alias (and its unused imports) in the builder tests in favor of the existing `BlockCache` alias. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-07 14:14:02 +02:00
5 people	d75f07b351	turbo-persistence: skip BlockCache for uncompressed (mmap-backed) blocks (#92390) ... ### What? Skip the `BlockCache` entirely for uncompressed (mmap-backed) SST blocks, and add a per-file CRC verification bitmap so checksums are verified at most once per file open for any block. ### Why? Uncompressed blocks produce an `ArcBytes` with `Backing::Mmap` — just an `Arc::clone` of the mmap handle plus a pointer into it. No heap allocation, no decompression. The cost of constructing one (an atomic refcount increment + pointer copy) is far less than a `BlockCache` hash + concurrent-map lookup + potential insertion. Bypassing the cache for these blocks: - Reduces lock contention on the shared `BlockCache` - Avoids polluting the cache with entries that are essentially free to recreate - Eliminates redundant mmap reads (the old code peeked at the header, then re-read it inside `read_block_generic`) Key blocks (index block + all key blocks) tend to be small and are frequently accessed. In practice, small key blocks are often stored uncompressed, so this optimization applies on the hot path. ### How? **Single mmap read per block access.** `get_or_cache_block` calls `get_raw_block_slice` exactly once to read the block header (with all `#[cfg(feature = "strict_checks")]` bounds guards), then branches on `uncompressed_length`: - `== 0` (uncompressed): verify CRC via bitmap, return mmap-backed `ArcBytes` directly - `> 0` (compressed): check `BlockCache`; on miss, verify CRC via bitmap, decompress, insert **CRC verification bitmap.** `StaticSortedFile` now holds a `Box<[AtomicU64]>` with one bit per block, set once the block's CRC has been verified. This prevents re-computing the checksum on every access for uncompressed blocks (which bypass the cache) and for compressed blocks that were evicted and re-read. `Relaxed` ordering suffices since racing first-time verifications are idempotent. **`ArcBlockCacheReader` wrapper.** A `Copy` struct bundling `&BlockCache` + `&[AtomicU64]` that implements `ValueBlockCache<ArcBytes>`, threading the bitmap through the existing generic lookup machinery without changing the trait signature. **`BlockWeighter` safety net.** The mmap-backed branch now has a `debug_assert!(!val.is_mmap_backed())` to catch accidental insertion, with a negligible weight (64) as a defensive fallback in release builds. <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Luke Sandberg <[email protected]>	2026-04-07 07:01:57 +02:00
4 people	81d5e070f4	next-core: deduplicate output assets and detect content conflicts on emit (#92292) ... ### What? Adds deduplication and conflict detection to the asset emission stage in `crates/next-core/src/emit.rs`, and a new `IssueStage::Emit` variant in `turbopack-core`. Before emitting, assets are grouped by their output path. If multiple assets map to the same path: - If their content is identical, one is silently chosen (deduplication). - If their content differs, both versions are written to `<node_root>/<content_hash>.<ext>` and an `EmitConflictIssue` is raised for each conflict. All assets are still emitted — conflicts do not abort the build. ### Why? Previously, duplicate output assets for the same path were emitted unconditionally — whichever write happened last silently won. This masked build graph bugs where two different modules produced conflicting output files. Reporting conflicts as issues (rather than silently overwriting) makes them visible and easy to diagnose without breaking the build. ### How? - Collect all assets with their resolved paths via `try_flat_join`. - Bucket them into two `FxIndexMap<FileSystemPath, Vec<ResolvedVc<Box<dyn OutputAsset>>>>` — one for node-root assets and one for client assets. - For each bucket entry, call `check_duplicates`: compare every asset against the first using `assets_diff`. If content differs, emit an `EmitConflictIssue` as a turbo-tasks collectible — but still return the first asset so emission continues. - `assets_diff` is a `#[turbo_tasks::function]` that takes only `(asset1, asset2, extension, node_root)` — the `asset_path` stays out of the task key to avoid unnecessary task cardinality. When file content differs, it hashes each version with xxh3, writes them to `<node_root>/<hash>.<ext>`, and returns the paths in the detail message so the user can diff them. - `EmitConflictIssue` implements the `Issue` trait with `IssueStage::Emit` (new variant added to `turbopack-core`), `IssueSeverity::Error`, a descriptive title, and a detail message explaining the type of conflict. - Node-root and client assets are emitted in parallel via `futures::join!` (not `try_join!`) to ensure deterministic error reporting — both branches always run to completion so errors are reported in a consistent order. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-05 20:37:58 +02:00
4 people	f65b10a54d	turbo-tasks: replace async resolve fns with custom Future types (ResolveRawVcFuture, ResolveVcFuture, ToResolvedVcFuture) (#91554) ... ### What? Replace the `async fn resolve()`, `async fn resolve_strongly_consistent()`, and `async fn to_resolved()` methods on `RawVc`, `Vc<T>`, and `OperationVc<T>` with hand-written custom `Future` implementations, following the existing `ReadRawVcFuture` pattern. New types: - **`ResolveRawVcFuture`** (`raw_vc.rs`) — core implementation, replaces `async fn resolve_inner()` - **`ResolveVcFuture<T>`** (`vc/mod.rs`) — typed wrapper over `ResolveRawVcFuture`, returned by `Vc::resolve()` - **`ResolveOperationVcFuture<T>`** (`vc/operation.rs`) — typed wrapper, returned by `OperationVc::resolve()` - **`ToResolvedVcFuture<T>`** (`vc/mod.rs`) — typed wrapper, returned by `Vc::to_resolved()` All new future types expose a `.strongly_consistent()` builder method, enabling `resolve_strongly_consistent()` to be replaced by `.resolve().strongly_consistent()` at call sites. `ReadRawVcFuture` is also updated to delegate its phase-1 resolve loop to `ResolveRawVcFuture` instead of duplicating the logic. `std::task::ready!` is used throughout to simplify poll implementations. Also adds `#[inline(never)]` to `ReadRawVcFuture::poll` and `ResolveRawVcFuture::poll` to avoid inlining large poll implementations into every await site. ### Why? Performance, binary size, and improved API ergonomics: - The hand-written `Future` pattern (already used by `ReadRawVcFuture`) gives the compiler more predictable, smaller code than the state machines generated for `async fn`. The `#[inline(never)]` attributes on `poll` prevent large poll bodies from being duplicated at every await site, which the async desugaring otherwise allows. - The new builder API (`.resolve().strongly_consistent()`) is more composable and removes the need for separate `_strongly_consistent` method variants, reducing the number of methods on `RawVc`/`Vc`/`OperationVc`. - Having `ReadRawVcFuture` delegate to `ResolveRawVcFuture` removes the duplicated resolve loop and ensures both paths stay in sync. ### How? - `ResolveRawVcFuture` stores `current: RawVc`, `read_output_options: ReadOutputOptions`, `strongly_consistent: bool`, and `listener: Option<EventListener>`. Its `poll` replicates the loop from the old `resolve_inner` using `try_read_task_output` / `try_read_local_output`. - On `Err(listener)` from a `try_*` call, the listener is stored in `self.listener` and `Poll::Pending` is returned. At the top of the loop, `ready!(poll_listener(...))` re-polls it and short-circuits if still pending. - Consistency is downgraded to `Eventual` after the first `TaskOutput` hop, matching the previous behavior. - `strongly_consistent: true` keeps the `SUPPRESS_EVENTUAL_CONSISTENCY_TOP_LEVEL_TASK_CHECK` suppression across all polls (same logic as `ReadRawVcFuture`). - `ReadRawVcFuture` now holds a `ResolveRawVcFuture` for phase 1 and drives it via `Pin::new(&mut self.resolve).poll(cx)` before proceeding to the cell read in phase 2. This eliminates the duplicated loop that previously existed in both types. - Typed wrappers (`ResolveVcFuture<T>`, `ResolveOperationVcFuture<T>`, `ToResolvedVcFuture<T>`) delegate `poll` to the inner `ResolveRawVcFuture` and map the output to the appropriate typed result. - `OperationVc::resolve_strongly_consistent()` is removed; 16 call sites updated to `.resolve().strongly_consistent()`. - All new types implement `Unpin` and are exported from `lib.rs`. - `std::task::ready!` is used in all `poll` implementations to reduce boilerplate. No behavioral changes — this is a pure implementation refactor. ### Binary size impact A release build (`pnpm swc-build-native --release`) was measured before and after the branch changes on the same merge-base commit (`a41bef94`): | | Size | |---|---| | Base (`a41bef94`, before branch) | 199,690,656 bytes (~190.4 MB) | | Branch (`6f7846f9`, after changes) | 199,252,384 bytes (~190.0 MB) | | **Difference** | **−438,272 bytes (−428 KB, −0.22%)** | The branch produces a slightly smaller binary. The reduction comes primarily from the `#[inline(never)]` attributes preventing large `poll` bodies from being duplicated at every await site. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-04 08:11:57 +02:00
4 people	41839289de	Add MCP get_compilation_issues tool and NAPI method (#92062) ... ### What? Adds a new \`get_compilation_issues\` MCP tool (and the underlying \`projectGetAllCompilationIssues\` NAPI method) that returns all compilation issues from all routes in a single call. **New files:** - \`crates/next-napi-bindings/src/next_api/project.rs\` — \`project_get_all_compilation_issues\` NAPI function + \`get_all_compilation_issues_operation\` / \`get_all_compilation_issues_inner_operation\` turbo-tasks operations - \`packages/next/src/server/mcp/tools/get-compilation-issues.ts\` — MCP tool implementation - \`packages/next/src/server/mcp/tools/utils/format-compilation-issues.ts\` — output formatter - \`test/development/mcp-server/mcp-server-get-compilation-issues.test.ts\` — e2e test - \`test/development/mcp-server/fixtures/compilation-errors-app/\` — fixture app with three routes (valid page, missing-module error, syntax error) **Modified files:** - \`packages/next/src/build/swc/generated-native.d.ts\` + \`types.ts\` + \`index.ts\` — expose the new NAPI method - \`packages/next/src/server/mcp/get-or-create-mcp-server.ts\` — register the new tool, accept \`getTurbopackProject\` option - \`packages/next/src/server/dev/hot-reloader-turbopack.ts\` — pass \`project\` to MCP middleware - \`packages/next/src/telemetry/events/build.ts\` — add \`'mcp/get_compilation_issues'\` to the \`McpToolName\` union ### Why? The existing MCP tools require a browser session (and thus a specific route to be rendered) to surface compilation errors. \`get_compilation_issues\` works without a browser session and covers all routes proactively — useful for AI coding agents that want to check for errors across the whole app before trying to render a page. ### How? **NAPI layer:** \`project_get_all_compilation_issues\` calls a two-level turbo-tasks operation pair: 1. \`get_all_compilation_issues_inner_operation\` — iterates all endpoint groups via \`project.get_all_endpoint_groups(false)\` and calls \`endpoint_group.module_graphs().as_side_effect()\` on each. This builds the module graph (resolution + transformation) for every entrypoint without chunking, emitting, or code generation. Issues are emitted as turbo-tasks collectables. 2. \`get_all_compilation_issues_operation\` — wraps the inner op in \`strongly_consistent_catch_collectables\` to harvest issues/diagnostics/effects, then returns an \`OperationResult\`. > **Why not \`project.whole_app_module_graphs()\`?** > \`whole_app_module_graphs()\` calls \`drop_issues()\` in development mode to prevent every per-route HMR subscription from seeing all global issues. Calling it here would return zero issues in dev. Per-endpoint \`module_graphs()\` calls don't have this suppression. **Output formatting (\`format-compilation-issues.ts\`):** The raw Turbopack wire types are transformed before being returned to MCP consumers: - **StyledString → plain string**: \`title\`, \`description\`, and \`detail\` are \`StyledString\` union trees (recursive \`{type, value}\` objects). These are flattened to plain strings — \`text\`/\`code\`/\`strong\` variants return their \`.value\` directly, \`line\` joins with \`""\`, \`stack\` joins with \`"\n"\`. - **ANSI codes stripped**: \`codeFrame\` is a pre-rendered string from the Rust NAPI layer that contains ANSI terminal colour codes. These are stripped via \`next/dist/compiled/strip-ansi\`. - **1-indexed source positions**: Turbopack's \`SourcePos\` is 0-indexed for both \`line\` and \`column\`. The formatter adds \`+1\` to each so consumers get the conventional editor-style 1-indexed values. - **Deduplication**: the same issue can surface from multiple endpoints during the module graph traversal. Issues are deduplicated by a \`severity|filePath|title|startLine:startCol\` key. - **No diagnostics**: Turbopack diagnostics are internal telemetry (\`EVENT_BUILD_FEATURE_USAGE\` feature-adoption counters). They are not actionable for the user, so the \`diagnostics\` field is omitted entirely from the response. **MCP tool (\`get_compilation_issues\`):** Calls \`project.getAllCompilationIssues()\`, passes the issues through \`formatCompilationIssues()\`, and returns \`{ issues: FormattedIssue[] }\` as JSON. Works without a browser session. **Test:** A fixture app with three routes (valid \`app/page.tsx\`, \`app/missing-module/page.tsx\` importing a non-existent package, \`app/syntax-error/page.tsx\` with unclosed JSX) is used to verify: 1. The tool returns a result without any browser session 2. Module-not-found errors are detected 3. Syntax errors are detected 4. Issues include \`severity\`, \`title\`, \`filePath\` metadata — all as plain strings (not StyledString objects) e2e tests: added — \`test/development/mcp-server/mcp-server-get-compilation-issues.test.ts\` --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-03 14:38:44 +02:00
4 people	be3ee02b83	turbo-tasks-backend: stability fixes for task cancellation and error handling (#92254) ... ### What? Bug fixes and a refactoring in `turbo-tasks-backend` targeting stability issues that surface when filesystem caching is enabled: 1. **Preserve `cell_type_max_index` on task error** — when a task fails partway through execution, `cell_counters` only reflects the partially-executed state. Previously, `cell_type_max_index` was updated from these incomplete counters, which removed entries for cell types not yet encountered. This caused `"Cell no longer exists"` hard errors for tasks that still held dependencies on those cells. The fix skips the `cell_type_max_index` update on error, keeping it consistent with the preserved cell data (which already wasn't cleared on error). This bug manifested specifically with `serialization = "hash"` cell types (e.g. `FileContent`), where cell data is transient and readers fall back to `cell_type_max_index` to decide whether to schedule recomputation. 2. **Fix shutdown hang and cache poisoning for cancelled tasks** — three related fixes for tasks cancelled during shutdown: - `task_execution_canceled` now drains and notifies all `InProgressCellState` events, preventing `stop_and_wait` from hanging on foreground jobs waiting on cells that will never be filled. - `try_read_task_cell` bails early (before calling `listen_to_cell`) when a task is in `Canceled` state, avoiding pointless listener registrations that would never resolve. - Cancelled tasks are marked as session-dependent dirty, preventing cache poisoning where `"was canceled"` errors get persisted as task output and break subsequent builds. The session-dependent dirty flag causes the task to re-execute in the next session, invalidating stale dependents. 3. **Extract `update_dirty_state` helper on `TaskGuard`** — the "read old dirty state → apply new state → propagate via `ComputeDirtyAndCleanUpdate`" pattern was duplicated between `task_execution_canceled` and `task_execution_completed_finish`. The new `update_dirty_state` default method on `TaskGuard` handles both transitions (to `SessionDependent` or to `None`) and returns the aggregation job + `ComputeDirtyAndCleanUpdateResult` for callers that need post-processing (e.g. firing the `all_clean_event`). ### Why? These bugs caused observable failures when using Turbopack with filesystem caching (`--cache` / persistent cache): - `"Cell no longer exists"` panics/errors on incremental rebuilds after a task error. - Hangs on `stop_and_wait` during dev server shutdown. - Stale `"was canceled"` errors persisted in the cache breaking subsequent builds until the cache is cleared. ### How? Changes are in `turbopack/crates/turbo-tasks-backend/src/backend/`: **`mod.rs`:** - Guard the `cell_type_max_index` update block inside `if result.is_ok()` to skip it on error, with a cross-reference comment to `task_execution_completed_cleanup` (which similarly skips cell data removal on error — the two must stay in sync). - Move the `is_cancelled` bail in `try_read_task_cell` before the `listen_to_cell` call to avoid inserting phantom `InProgressCellState` events that would never be notified. - In `task_execution_canceled`: switch to `TaskDataCategory::All` (needed for dirty state metadata access), notify all pending in-progress cell events, and mark the task as `SessionDependent` dirty via the new helper. - In `task_execution_completed_finish`: replace ~77 lines of inline dirty state logic with a call to `task.update_dirty_state(new_dirtyness)`, preserving the `all_clean_event` post-processing and the `dirty_changed` variable under `#[cfg(feature = "verify_determinism")]`. **`operation/mod.rs`:** - Add `update_dirty_state` default method on `TaskGuard` trait (~60 lines), co-located with the existing `dirty_state()` reader. Takes `Option<Dirtyness>`, applies the transition, builds `ComputeDirtyAndCleanUpdate`, and returns `(Option<AggregationUpdateJob>, ComputeDirtyAndCleanUpdateResult)`. - Add `ComputeDirtyAndCleanUpdateResult` to the public re-exports. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-03 10:05:17 +02:00
4 people	e464ca398c	Fix pr-status.js: retry on API errors and report timed-out jobs (#92205) ... ### What? Fixes two bugs in `scripts/pr-status.js` that caused the script to silently report zero failures when there were actual CI failures. ### Why? When running `node scripts/pr-status.js 92080`, the script reported "No failed jobs found" despite the PR having real failures. Two root causes: 1. **Transient API errors silently swallowed.** `getAllJobs()` had a bare `catch { break }` that returned an empty array on any GitHub API error (e.g., HTTP 502). Since the GitHub Actions jobs API for large runs frequently returns transient 502s, this caused false "no failures" reports. 2. **`timed_out` and `startup_failure` jobs were invisible.** The script only checked for `conclusion === 'failure'`, but GitHub uses distinct conclusion values like `timed_out` (job exceeded timeout) and `startup_failure` (runner failed to start). These jobs fell through all filters and were silently omitted from reports. ### How? **Retry logic in `getAllJobs()`:** - Retries each paginated API call up to 3 times with 2s/4s backoff - If all retries fail on the first page, throws an error (no silent empty results) - If later pages fail after partial data is collected, warns and returns what was fetched **Broader failure detection with `FAILED_CONCLUSIONS`:** - Added a shared `FAILED_CONCLUSIONS` set: `{'failure', 'timed_out', 'startup_failure'}` - Used in `getFailedJobs()`, `categorizeJobs()`, and flaky test detection - Jobs with non-`failure` conclusions are annotated in the report table (e.g., "(timed_out)") - `getFailedJobs()` now also returns the `conclusion` field so downstream code can distinguish failure types Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-04-02 13:22:13 +02:00
Tobias KoppersandGitHub	9e1509307e	Fix DashMap read-write self-deadlock in task_cache causing hangs (#92210)	2026-04-01 13:25:03 -07:00
5 people	3e0158846e	feat: add NEXT_HASH_SALT env var for content-hash filename salting (#91871) ... ### What? Adds a `NEXT_HASH_SALT` environment variable **and** a `experimental.outputHashSalt` config option that mix a user-supplied string into every content-addressed hash used to generate chunk filenames and static asset filenames. This works for both Webpack and Turbopack. When both are set, the values are concatenated (`outputHashSalt + NEXT_HASH_SALT`), so a per-project salt can be baked into `next.config.js` while a per-deployment salt is injected at build time via the environment variable. ### Why? Content-addressed filenames (e.g. `chunk.abc123.js`) are derived from file content, so they only change when the content changes. There are deployment scenarios where you need to force all filenames to rotate — for example after a CDN misconfiguration has poisoned caches for a particular hash space — without actually changing source code. A stable, opt-in salt lets operators do this without touching application code. Some customers prefer the config-file approach (`turbopack.outputHashSalt`) over environment variables, so both are supported. ### How? **Webpack** already has `output.hashSalt` in its config. We simply forward `NEXT_HASH_SALT` to that option. **Turbopack** required threading the value through several layers: 1. The effective hash salt is computed once in `assignDefaultsAndValidate` as `config.turbopackHashSalt = (turbopack.outputHashSalt ?? '') + (NEXT_HASH_SALT ?? '')` and stored on `NextConfigComplete`. Both `turbopackBuild` (production) and `createHotReloaderTurbopack` (dev) read from this single field. 2. `ProjectOptions.hash_salt` receives the pre-computed salt. 3. `Project` stores the salt and passes it into the three chunking context option structs (`ClientChunkingContextOptions`, `ServerChunkingContextOptions`, `EdgeChunkingContextOptions`). 4. Both `BrowserChunkingContext` and `NodeJsChunkingContext` gain a `hash_salt: RcStr` field. 5. A new `deterministic_hash_with_salt(salt, input, algorithm)` function in `turbo-tasks-hash` writes the salt bytes first, then the content bytes, into a single hasher — one pass, no hash-of-hash composition. 6. A matching `content_hash_with_salt` method is added to `FileContent` and `AssetContent`. 7. `ChunkingContext::asset_path` is changed to accept `Vc<AssetContent>` (instead of a pre-computed `Vc<RcStr>`) so the chunking context can choose the correct hash path itself. `StaticOutputAsset::path` simplifies accordingly. Without `NEXT_HASH_SALT` and without `turbopack.outputHashSalt` set, behaviour is identical to before — no hash change, no performance impact. **e2e test** (`test/production/app-dir/hash-salt/`) verifies: - Two builds with the same salt produce identical chunk and static asset filenames. - A build with a different salt produces different filenames. - `turbopack.outputHashSalt` (config) changes filenames vs no salt. - Combined config + env salt differs from either alone. - Runs for both Turbopack and Webpack. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Luke Sandberg <[email protected]>	2026-04-01 22:03:50 +02:00
4 people	d84e3bb56e	Use RcStr directly in napi(object) structs instead of converting to String (#92014) ... ### What? Replace `String` fields in `#[napi(object)]` structs within `crates/next-napi-bindings/` with `RcStr`, and remove the corresponding unnecessary `.to_string()` / `.into_owned()` conversions at the NAPI boundary. Affected structs and fields: | File | Struct | Fields | |---|---|---| | `utils.rs` | `NapiIssue` | `file_path`, `documentation_link` | | `utils.rs` | `NapiAdditionalIssueSource` | `description` | | `utils.rs` | `NapiSource` | `ident`, `file_path` | | `utils.rs` | `NapiDiagnostic` | `category`, `name`, `payload` | | `project.rs` | `NapiRoute` | `pathname` | | `endpoint.rs` | `NapiAssetPath` | `path`, `content_hash` | ### Why? `RcStr` already implements `ToNapiValue` and `FromNapiValue` (behind the `napi` feature flag in `turbo-rcstr`, already enabled in `next-napi-bindings`). Despite this, many `#[napi(object)]` structs were still declared with `String` fields and converted from `RcStr` source data via `.to_string()` or `.into_owned()`, allocating a new heap `String` on every call just to hand it straight to the NAPI serializer. Since `RcStr` is reference-counted, using it directly avoids the allocation entirely — the refcount is simply incremented on clone and the underlying bytes are passed to `napi_create_string_utf8` directly. ### How? - Changed `String` → `RcStr` in the struct field declarations. - Updated the corresponding `From` / constructor impls to use `.clone()` instead of `.to_string()` / `.into_owned()` on `RcStr` values, and direct field moves where ownership is transferred. - Added `use turbo_rcstr::RcStr;` imports where not already present (`utils.rs`, `endpoint.rs`). `project.rs` already imported `RcStr`. - The JavaScript API surface is unchanged: both `String` and `RcStr` serialize to a JS `string` via the same underlying `napi_create_string_utf8` call. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-28 14:18:14 -07:00
4 people	63f485248b	test(interoperability): increase waitForElementByCss timeout for cross-router navigations (#91918) ... ### What? Increase the `waitForElementByCss` timeout from the default (~10 s) to 30 s on every cross-router navigation assertion in `test/e2e/app-dir/interoperability-with-pages/navigation.test.ts`. ### Why? [Datadog test runs](https://app.datadoghq.com/ci/test/runs?query=test_level%3Atest%20%40test.source.file%3Atest%2Fe2e%2Fapp-dir%2Finteroperability-with-pages%2Fnavigation.test.ts%20%40test.status%3Afail%20%40test.name%3A%22navigation%20between%20pages%20and%20app%20dir%20It%20should%20be%20able%20to%20navigate%20pages%20-%3E%20app%20and%20go%20back%20an%20forward%22&agg_m=count&agg_m_source=base&agg_t=count&cols=%40test.status%2Ctimestamp%2C%40duration%2C%40test.type%2C%40test.is_known_flaky%2C%40test.name&currentTab=overview&eventStack=&fromUser=false&index=citest&start=1773951834834&end=1774556634834&paused=false) Navigating between the App Router and Pages Router (in either direction) always triggers a **full-page reload** because the two routers cannot share a client-side navigation. In dev mode, the destination page is compiled on-demand the first time it is visited, which consistently takes ~13 s in CI. The default `waitForElementByCss` timeout is ~10 s, so the `app -> pages` test would intermittently time out before the newly compiled page finished loading — a classic flaky-test failure that is load-sensitive and non-deterministic across machines. This is a test-only fix; no framework behaviour changes. ### How? Pass `{ timeout: 30000 }` as the second argument to every `waitForElementByCss` call that waits for an element after a cross-router navigation (8 call-sites across 4 tests). A comment is added before each group of calls explaining why the higher value is necessary. Verified locally with 3 consecutive runs — all 4 tests pass, and the previously flaky `app -> pages` case now completes in ~13 s without timing out. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-26 14:20:55 -07:00
4 people	ad65b1bdcf	test(app-basepath): increase waitForElementByCss timeout for hard-nav test (#91920) ... ### What? Increase the `waitForElementByCss('#page-2', ...)` timeout in the `app dir - basepath › should successfully hard navigate from pages -> app` test from the default 10 s to 30 s. ### Why? [Datadog test runs](https://app.datadoghq.com/ci/test/runs?query=test_level%3Atest%20%40test.source.file%3Atest%2Fe2e%2Fapp-dir%2Fapp-basepath%2Findex.test.ts%20%40test.name%3A%22app%20dir%20-%20basepath%20should%20successfully%20hard%20navigate%20from%20pages%20-%3E%20app%22&agg_m=count&agg_m_source=base&agg_t=count&cols=%40test.status%2Ctimestamp%2C%40duration%2C%40test.type%2C%40test.is_known_flaky%2C%40test.name&currentTab=overview&eventStack=&fromUser=false&index=citest&start=1773951667400&end=1774556467400&paused=false) The test navigates from a Pages Router page (`/base/pages-path`) to an App Router page (`/base/another`) via a hard (non-client-side) navigation. In dev mode, the destination App Router page is compiled on demand the first time it is visited. That compilation can take longer than the default 10 s `waitForElementByCss` timeout, causing the test to fail spuriously with: ``` page.waitForSelector: Timeout 10000ms exceeded ``` ### How? Pass `30000` (30 s) as the second argument to `waitForElementByCss`. This matches the pattern used elsewhere in the test suite for dev-mode tests that are subject to on-demand compilation latency. No runtime code changes are needed — this is purely a test-side fix. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-26 14:20:37 -07:00
4 people	f7d487057c	test(typed-routes): fix flaky tests by increasing retry timeout (#91923) ... ## Fixing a bug ### What? Three tests in `test/e2e/app-dir/typed-routes/typed-routes.test.ts` were intermittently failing with `Failed to retry within 3000ms`: - `typed-routes › should generate route types correctly` - `typed-routes › should have passing tsc after start` - `typed-routes › should correctly convert custom route patterns from path-to-regexp to bracket syntax` ### Why? The test harness marks the dev server as "ready" when it sees `✓ Ready in X` in stdout. However, the server logs this message _before_ completing the full initialization sequence in `start-server.ts` — specifically before `getRequestHandlers()` resolves, which chains into `setupDevBundler()`. Route type generation (`routes.d.ts`) only completes inside `setupDevBundler`'s Watchpack `aggregated` event handler. The gap between the "Ready" log and actual `routes.d.ts` completion can exceed 3000ms on loaded CI runners, causing the tests' `retry()` calls to time out before the file exists. A secondary issue in `should have passing tsc after start`: the test called `next.stop()` immediately after the server was considered ready, before `routes.d.ts` was written. This meant `tsc` ran against a project missing the generated types, causing type errors even though the server had been healthy. ### How? - Increased the `retry()` timeout from the default 3000ms to 30000ms for the three affected tests. - Added a `retry()` guard in `should have passing tsc after start` to wait for `routes.d.ts` to be present and correct _before_ stopping the server. This ensures `tsc` sees complete generated types. Verified by running the full test file 3 consecutive times locally with `NEXT_SKIP_ISOLATE=1 NEXT_TEST_MODE=dev pnpm testheadless` — all passes, all 5 tests green each run. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-26 14:20:13 -07:00
5 people	0090db224d	turbo-tasks: add hashed cell mode for hash-based change detection without cell data (#91576) ... ## Summary ### Core: hash-based cell change detection - Adds `cell_data_hash: AutoMap<CellId, [u8; 16]>` to the storage schema (`category = "data"`) so the backend can persist a hash of transient cell data across evictions - Stored as `[u8; 16]` (little-endian bytes of a u128) rather than `u128` to keep the 1-byte alignment out of `AutoMap` and therefore out of the `LazyField` enum — a bare `u128` would grow the enum from 56 to 64 bytes due to its 16-byte alignment requirement - Adds `content_hash: Option<u128>` to `UpdateCellOperation::run` and threads it through the full call chain: `CurrentCellRef` → `TurboTasksCallApi::update_own_task_cell` → `Backend::update_task_cell` → operation - **New invalidation logic** in `UpdateCellOperation::run` for the `assume_unchanged = false` path: - Old content available → real equality compare (unchanged) - Old content evicted, hashes match → write new content but **skip dependent invalidation** - Old content evicted, hashes differ (or missing) → write + invalidate as before - `cell_data_hash` is updated whenever content is written (skipped if hash is unchanged; always updated for non-serializable cells regardless of `assume_unchanged`) - Adds `hashed_compare_and_update` / `hashed_compare_and_update_with_shared_reference` methods to `CurrentCellRef` (require `T: PartialEq + DeterministicHash`); hash is computed lazily (only after equality check fails when old value is available) - Adds `VcCellHashedCompareMode<T>` in `cell_mode.rs` which implements `VcCellMode<T>` for `T: VcValueType + PartialEq + DeterministicHash` ### Macro: `serialization = "hash"` and `hash = "manual"` - **`serialization = "hash"`** — behaves like `serialization = "none"` (no disk serialization, transient) but uses `VcCellHashedCompareMode` so the stored hash prevents spurious downstream invalidation when transient data is evicted and re-executed - Valid only with `cell = "compare"` (the default); combining with `cell = "new"` or `cell = "keyed"` is a compile error - Automatically derives `DeterministicHash` on the annotated type - **`hash = "manual"`** — opt-out of the auto-derive when a custom `DeterministicHash` impl is needed (analogous to `eq = "manual"`); using `hash = "manual"` without `serialization = "hash"` is a compile error ### turbo-tasks-fs: `PersistedFileContent` - Adds `PersistedFileContent` — a mirror of `FileContent` that is returned by `Vc<FileContent>::persist()`, storing the same data but obtained through the persistent cache path - `FileContent` is switched to `serialization = "hash"` so the macro auto-derives `DeterministicHash` - `DiskFileSystem::write()` now calls `.persist().await?` before emitting the write effect, using `PersistedFileContent` for the file comparison and write — this ensures full content is in the persistent cache on restore, avoiding spurious downstream invalidation - `WriteContent::File` in `invalidator_map.rs` is updated to hold `ReadRef<PersistedFileContent>` ### turbopack-core: `Code` with hash-based serialization - Adds `DeterministicHash` impls for `SmallVec<[T; N]>` and `()` in `turbo-tasks-hash` - Switches `Code` to `serialization = "hash"` so module factory code cells use hash-based change detection - Calls `.persist()` on module factory code cells in `turbopack-ecmascript` ### next-core: duplicate asset detection and parallel emit - `emit_assets` now emits node and client assets concurrently via `try_join!` - Detects duplicate assets (same output path, different content) and returns a hard error with a diff summary — previously the last writer silently won ## Macro syntax ```rust #[turbo_tasks::value(serialization = "hash")] struct MyTransientType { ... } // opt out of auto-derived DeterministicHash when you need a custom impl: #[turbo_tasks::value(serialization = "hash", hash = "manual")] struct MyTransientType { ... } ``` ## Test plan - [x] Integration tests in `turbopack/crates/turbo-tasks-backend/tests/hashed_cell_mode.rs`: - `test_hashed_cell_mode_change_triggers_invalidation` — value change triggers consumer re-execution - `test_hashed_cell_mode_equal_value_no_invalidation` — same hash prevents consumer re-execution - [x] `cargo test -p turbo-tasks-macros-tests` passes - [x] `cargo test -p turbo-tasks-backend --test hashed_cell_mode` passes - [x] `cargo check --workspace` passes --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>	2026-03-26 12:32:45 -07:00
4 people	548fbca6df	turbopack: Remove Vc::resolve(), migrate all callsites to Vc::to_resolved() (#91725) ... ### What? Remove the `Vc::resolve()` method and migrate all ~37 callsites across the turbopack and next crates to use `Vc::to_resolved()` instead. ### Why? `Vc::resolve()` returned `Result<Vc<T>>` — a resolved `Vc` with no static guarantee that it is resolved. This is a footgun: the type is indistinguishable from any other unresolved `Vc<T>`, so it's easy to accidentally lose the "resolved" property at a type boundary or store it in a struct without the compiler enforcing invariants. `Vc::to_resolved()` returns `Result<ResolvedVc<T>>`, which statically encodes the resolved guarantee in the type. `ResolvedVc<T>` implements `Deref<Target = Vc<T>>`, so it coerces seamlessly to `Vc<T>` at call sites that still need the plain type. Removing `Vc::resolve()` eliminates this footgun entirely and pushes callers toward the type-safe API. ### How? - Removed the `Vc::resolve()` method from `turbo-tasks/src/vc/mod.rs`. - Updated `Vc::to_resolved()` to construct the inner `Vc` directly via `self.node.resolve()` rather than calling `self.resolve()`. - Updated `Vc::debug_identifier` to go through `to_resolved()`. - Updated `TaskInput::resolve_input` for `Vc<T>` to use `to_resolved()`. - Updated all callsites across turbopack-core, turbopack-ecmascript, turbopack-resolve, turbopack-browser, turbopack-css, turbopack-node, turbopack-ecmascript-runtime, next-api, next-core, and test/fuzz files: - Where `ResolvedVc<T>` is usable (deref coercion handles method calls, or the value is stored in a `HashMap` etc.), simply replaced `.resolve()` with `.to_resolved()`. - Where a plain `Vc<T>` is strictly required by a callee signature, dereferenced with `*x.to_resolved().await?`. - Updated the `AfterResolvePluginWithCondition` type alias in `resolve/mod.rs` to use `ResolvedVc<AfterResolvePluginCondition>` now that the tuple member is produced by `to_resolved()`. - Updated doc comments referencing `Vc::resolve` to point to `Vc::to_resolved`. Verified with `cargo check --all-targets` — compiles cleanly with no errors. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-26 11:30:36 -07:00
4 people	73e654f678	fix(turbopack): preserve resolveExtensions priority in read_matches fast path (#91856) ... ### What? Fix a bug where Turbopack ignored the priority order of `resolveExtensions` when both a platform-specific variant (e.g. `Component.web.tsx`) and a default variant (`Component.tsx`) exist for the same module path. The first-listed extension should always win, but it often didn't. ### Why? Turbopack's `read_matches` function iterates over constant alternatives extracted from a resolve pattern (one per extension in `resolveExtensions`). It uses `enumerate()` to assign a priority `index` to each alternative — lower index = higher priority — so the correct file is picked after sorting. The `until_end=true` branch (direct file existence check) correctly threaded `index` through. But the `until_end=false` branch (subdirectory descent, the path taken when the pattern contains a `/` before the dynamic suffix) had a **hardcoded `0`** instead of `index`: ```rust // before — all alternatives get the same priority nested.push((0, read_matches(fs_path.clone(), ...))); // after — priority matches the extension's position in resolveExtensions nested.push((index, read_matches(fs_path.clone(), ...))); ``` With all alternatives at priority 0, the tie-breaking sort was effectively arbitrary. In practice `Component.tsx` frequently beat `Component.web.tsx` even when `.web.tsx` was listed first, breaking platform-specific module overrides — a common React Native Web / Expo Web pattern. ### How? **One-line Rust fix** in `turbopack/crates/turbopack-core/src/resolve/pattern.rs`: replace the hardcoded `0` with the loop variable `index` in the `until_end=false` branch of the fast path. **Turbopack unit tests** (`turbopack-core`): - `test_custom_extensions_web_before_default` — `.web.tsx` wins when listed first in `resolveExtensions` - `test_custom_extensions_fallback_when_web_missing` — correctly falls back to `.tsx` when `.web.tsx` is absent - Extended `test_read_matches` with `extension_ordering` and `subpath_ordering` cases covering both the `until_end=true` and `until_end=false` branches **Next.js e2e test** (extended `test/e2e/app-dir/resolve-extensions/`): - Added `PlatformComponent.web.tsx` (renders `"hello web platform"`) and `PlatformComponent.tsx` (renders `"hello default platform"`) - `.web.tsx` is placed before `.tsx` in `resolveExtensions` in `next.config.js` (for both Turbopack and webpack) - New assertion: SSR'd HTML must contain `"hello web platform"` and must **not** contain `"hello default platform"` Fixes #91117 - [x] Tests added. See: https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-26 11:28:06 -07:00
Tobias KoppersandGitHub	16935744fb	turbo-tasks-backend: assert non-transient task_ids in track_modification (#91924) ... ### What? Add a `debug_assert!` in `StorageWriteGuard::track_modification` to enforce that transient `TaskId`s are never inserted into the persistence-modified set. ### Why? Concurrent server component HMR updates triggered a runtime panic in Turbopack: ``` internal error: entered unreachable code: transient task_ids should never be enqueued to be persisted ``` This `unreachable!` lives in `snapshot_and_persist` (`backend/mod.rs:1254`), which is called during the persist cycle. It fires when a transient `TaskId` is found in `Storage::modified` — the set that tracks tasks whose state needs to be written to disk. **Transient task IDs (high bit set) are never serialized.** The invariant was enforced only at the *caller layer*: - `TaskGuardImpl::track_modification` (`operation/mod.rs:1029`) guards with `if !self.task_id.is_transient()` - `TaskGuardImpl::invalidate_serialization` (`operation/mod.rs:970`) guards with `if !self.task_id.is_transient()` But `StorageWriteGuard::track_modification` — the public method on the storage struct itself — had no such guard. Under concurrent HMR invalidations, a transient task ID could reach this method via a code path that bypasses the caller-level checks, inserting the ID into `Storage::modified` and causing the panic downstream during the next persist cycle. The flaky test is `test/development/app-dir/hmr-iframe/hmr-iframe.test.ts` — it triggers two simultaneous server component file changes (one in an iframe, one in the parent), which races the persist cycle against concurrent invalidations. ### How? Add a `debug_assert!` in `StorageWriteGuard::track_modification` (the sole public entry point into the storage mutation path) that matches the invariant already checked at the caller layer: ```rust debug_assert!( !self.inner.key().is_transient(), "transient task_ids should never be enqueued to be persisted" ); ``` This enforces the invariant at the storage boundary in debug builds, surfacing the violation at the point of insertion rather than much later during the persist cycle. The message deliberately matches the existing `unreachable!` downstream so both failure modes are easy to correlate. **Verification:** - `cargo test -p turbo-tasks-backend` — all unit tests pass - `pnpm build-all` — build succeeds - `NEXT_SKIP_ISOLATE=1 NEXT_TEST_MODE=dev pnpm testheadless test/development/app-dir/hmr-iframe/hmr-iframe.test.ts` — 3/3 consecutive passes (was flaky before)	2026-03-25 22:59:20 -07:00
4 people	df88058f5e	Turbopack: switch from base40 to base38 hash encoding (remove ~ and . from charset) (#91832) ... ### What? Switch Turbopack's hash encoding charset from base40 (`0-9 a-z _ - ~ .`) to base38 (`0-9 a-z _ -`), removing the `~` and `.` characters. Pure rename/charset change — no structural changes. ### Why? The `~` and `.` characters in base40-encoded filenames are blocked by standard Nginx hardening rules (`block_common_exploits.conf`) and enterprise WAF configurations, causing **403 Forbidden errors** when applications are deployed behind security-hardened infrastructure. Examples of problematic filenames: - `turbopack-0c3o1svijj_~~.js` — `~~` flagged as directory traversal / injection - `0...f7~att2_2.js` — `...` flagged as path traversal attempt - `0q~2copru0zy0.css` — `~` filtered by some WAF rulesets Previous hex-only filenames (e.g. `turbopack-01ca012029ca2e66.js`) had no such issues. Fixes https://github.com/vercel/next.js/issues/91678 ### How? **Charset change** (`turbo-tasks-hash/src/base38.rs`, renamed from `base40.rs`): - Alphabet reduced from 40 to 38 characters: `0123456789abcdefghijklmnopqrstuvwxyz_-` - All constants and functions renamed: `BASE40_*` → `BASE38_*`, `encode_base40` → `encode_base38` - Hash lengths unchanged: 13 chars for 64-bit, 25 chars for 128-bit (`38^13 > 2^64`, `38^25 > 2^128`) - Content hash length stays at 13 (68.2 bits vs 69.2 bits with base40 — negligible) **`HashAlgorithm` enum variants** (`turbo-tasks-hash/src/lib.rs`): - `Xxh3Hash64Base40` → `Xxh3Hash64Base38` - `Xxh3Hash128Base40` → `Xxh3Hash128Base38` **Bit computation comment** (`turbopack-core/src/ident.rs`): - Updated `7 base38 chars ≈ 37 bits` → `≈ 36 bits` (log2(38) × 7 = 36.7) **Test regex patterns** (15 test files): - Updated `[0-9a-z_.~-]` → `[0-9a-z_-]` to match the new charset --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-25 08:27:30 +01:00
4 people	78f1625686	turbo-tasks-backend: improve print_cache_item_size instrumentation (#91742) ... ### What? Fixes a compilation hang that occurred when the `print_cache_item_size` debug feature was enabled, and cleans up the surrounding instrumentation code. **Root cause of the hang (double lock / deadlock):** During `persist_snapshot`, the storage iterates over all modified tasks. For each task it calls: ```rust let inner = self.shard.storage.map.get(&task_id).unwrap(); // ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ // acquires a READ lock on the DashMap shard for task_id let item = (self.shard.process)(task_id, &inner, &mut self.buffer); ``` The `process` closure receives `&TaskStorage` while that **read lock is still held**. Inside the closure, the old code called: ```rust self.get_task_name(task_id, turbo_tasks) ``` `get_task_name` creates an `ExecuteContext`, then calls `ctx.task(task_id, TaskDataCategory::Data)`, which calls `storage.access_mut(task_id)`, which calls `self.map.entry(key)`. That `entry()` call tries to acquire a **write lock on the same DashMap shard** — which is already read-locked on the same thread. `parking_lot::RwLock` is not reentrant, so the thread blocks forever waiting for its own read lock to be released. **Fix:** the `&TaskStorage` reference (`inner`) is already in scope inside the closure. Call `inner.get_persistent_task_type()` directly instead of going through `get_task_name`, which avoids any additional lock acquisition. **Additional changes in this PR:** 1. **Make compressed-size reporting opt-in** — split `print_cache_item_size` into two Cargo features. The base feature now shows uncompressed sizes only (no lz4 overhead). A new `print_cache_item_size_with_compressed` feature re-enables compressed-size reporting for when you need it. 2. **Extract helpers to eliminate duplication** — the output block repeated `#[cfg]`-gated `FormatSizes { … }` struct literals six times and duplicated the `task_name` computation twice. Extract: - `FormatSizes` struct + `impl Display` (with `#[cfg]` isolated inside) - `TaskCacheStats::task_name(storage)` — single source for the grouping key - `TaskCacheStats::sort_key()` — encapsulates the `#[cfg]`-switched primary sort field - `TaskCacheStats::format_total/data/avg_data/meta/avg_meta()` — each returning `FormatSizes` ### Why? The compilation was hanging and unusable. The lock ordering bug was non-obvious because the re-entrant acquisition happened across an abstraction boundary (`get_task_name` hiding the `access_mut` call). ### How? Avoid the redundant lock by using the `&TaskStorage` already provided to the closure. Pure refactor for everything else — no behaviour change for any given feature combination. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-25 08:16:14 +01:00
4 people	43f9017719	Add next internal post-build CLI command for Turbopack database compaction (#91336) ... ### What? Adds a \`next internal post-build\` CLI command that performs Turbopack persistent cache database compaction as a separate step after \`next build\`. ### Why? Database compaction during shutdown can be slow and blocks the build process. By deferring compaction to a separate post-build step, we can: - Keep the main \`next build\` fast by skipping compaction at shutdown (controlled via \`NEXT_USE_POST_BUILD=1\`) - Run compaction independently, e.g. as a separate CI step or in the background - Perform a full compaction (\`max_merge_segment_count: usize::MAX\`) that is more thorough than the runtime shutdown compaction **Intended usage in a build pipeline:** \`\`\`sh NEXT_USE_POST_BUILD=1 next build # fast build, skips compaction at shutdown next internal post-build # separate step: full compaction of the cache \`\`\` ### How? **Rust side:** - Added \`compact_database()\` to \`turbo-tasks-backend\` that opens the database and runs a full compaction using \`TurboTasksParallelScheduler\` backed by a dedicated multi-threaded Tokio runtime (same scheduler as the normal runtime path, enabling parallel work-stealing) - Added \`turbopack_database_compact\` NAPI binding that resolves the versioned DB path and calls \`compact_database()\` - Added \`skip_compaction\` flag to \`NapiTurboEngineOptions\` / \`TurboKeyValueDatabase\` to skip compaction during normal shutdown when post-build will handle it; asserted that \`skip_compaction=true\` requires \`is_short_session=true\` (enforcing the logical dependency) - Extracted shared \`DB_CONFIG\`, \`FAMILIES\`, and \`COMPACT_CONFIG\` constants and \`git_version_info()\` helper to avoid duplication between the runtime DB and standalone compaction paths **TypeScript side:** - Added \`next internal post-build\` command that loads the Next.js config, finds the Turbopack cache directory, and calls the native compaction binding - Added \`databaseCompact\` to the turbo bindings interface (native + WASM stub) - \`turbopack-build/impl.ts\` sets \`skipCompaction: true\` when \`NEXT_USE_POST_BUILD=1\` **Tests:** - Added \`test/production/app-dir/post-build/\` integration test that verifies compaction runs successfully on a persistent-cache build and reports nothing to do when persistent caching is disabled --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-24 13:34:35 -07:00
4 people	7984e4ab5c	turbopack-cli: add --persistent-caching flag for filesystem-backed cache (#91657) ... ### What? Adds `--persistent-caching` and `--cache-dir` CLI flags to `turbopack dev` and `turbopack build` in the standalone `turbopack-cli` crate. ### Why? The standalone `turbopack-cli` binary (used independently of `next`) had no way to opt into the filesystem-backed persistent cache that the `next-napi-bindings` integration already supports. Without this, repeated builds always start cold — every incremental run re-computes the full task graph from scratch. ### How? **`build.rs` (new):** A Cargo build script that uses `vergen-gitcl` to emit `VERGEN_GIT_DESCRIBE` and `VERGEN_GIT_DIRTY` as compile-time env vars. These are used to key the cache by git version, matching the same approach as `next-napi-bindings`. The build script embeds the dirty state at compile time rather than runtime to avoid a git subprocess on every startup. The dirty-state caching tradeoff (explained in the build script comment): there's a narrow edge case where the tree becomes newly dirty without triggering a Cargo rebuild, but in practice this is safe — if nothing in the crate changed, the binary is semantically equivalent to a clean build, so using the HEAD hash as the cache key is correct. **`arguments.rs`:** Two new flags on `CommonArguments` (shared by both `dev` and `build` subcommands): - `--persistent-caching` — opt-in flag to enable the filesystem cache - `--cache-dir <PATH>` — override the cache directory (defaults to `.turbopack/cache` in the project root) **`dev/mod.rs` and `build/mod.rs`:** - The `Backend` type alias changes from `TurboTasksBackend<NoopBackingStorage>` to `TurboTasksBackend<Either<TurboBackingStorage, NoopBackingStorage>>` to unify the type without dynamic dispatch. - When `--persistent-caching` is set: initializes `TurboBackingStorage` via `turbo_backing_storage()`, selects `StorageMode::ReadWriteOnShutdown` for CI/short sessions and `StorageMode::ReadWrite` for interactive dev, and prints a warning if the cache was invalidated on startup. - When not set: falls back to `noop_backing_storage()` — identical behavior to before. - Respects the `TURBO_ENGINE_READ_ONLY` env var to force read-only mode. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-20 19:15:03 +01:00
5 people	7d451fe01e	Add module count field to module graph tracing spans (#91697) ... ### What? Adds a `modules` field to two tracing spans that cover module graph construction: - `"module graph for endpoint"` span (per-page graph, in `app.rs`) — records the total number of modules across all `SingleModuleGraph`s built for that endpoint. - `"whole app module graph"` span (new info-level span wrapping `whole_app_module_graph_operation` in `project.rs`) — records the combined module count of the base graph and the additional-entries graph. Also adds a `module_count()` method to `SingleModuleGraph` (as a `#[turbo_tasks::function]` returning `Vc<u64>`) that exposes `number_of_modules`. ### Why? Module graph size directly affects build and HMR performance. Without a count in the trace, it is impossible to correlate span duration with graph size or spot regressions where the graph grows unexpectedly. Surfacing the count as a structured span field makes it queryable without requiring log parsing. ### How? - `SingleModuleGraph::module_count()` is a cheap `#[turbo_tasks::function]` that wraps the already-computed `number_of_modules: usize` field as a `Vc<u64>`. - Both call sites follow the deferred-field pattern already used in `ProjectContainer::initialize`: the span is created with `modules = Empty` before the `async move` block, `span.record("modules", count)` is called once the graphs are ready, and `span_clone` is passed to `.instrument()` so the field is recorded on the correct span. - Module counts are summed across all graphs that contribute to the endpoint/app. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>	2026-03-20 19:14:48 +01:00
4 people	25646b928f	[turbopack] Remove incorrect debug_assert in try_read_task_cell (#91699) ... ### What? Removes the `debug_assert_not_in_top_level_task` call from `try_read_task_cell` in `turbopack/crates/turbo-tasks/src/manager.rs`. ### Why? The assertion was wrong. `debug_assert_not_in_top_level_task` panics with the message: > "Eventually consistent read cannot be performed from a top-level task." The restriction only applies to **eventually consistent** reads (e.g. task output reads), which can return stale data if called from a top-level task (`.run_once(...)`). Cell reads, however, are **strongly consistent** — they always return up-to-date data. There is no semantic reason to forbid reading a cell from a top-level task, and the assert was producing false positives in valid call sites. ### How? Delete the three lines that guard and invoke `debug_assert_not_in_top_level_task` inside `try_read_task_cell`. The function itself and its usage in `try_read_task_output` / `try_read_local_output` (which are genuinely eventually-consistent) are unchanged. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-20 19:14:00 +01:00
4 people	47230b12d4	turbopack: move "compact database" tracing span to backend layer (#91693) ... ### What? Move the `"compact database"` tracing span from `turbo-persistence` (`db.rs`) to the backend layer (`turbo-tasks-backend/src/backend/mod.rs`), make it a root span, group it with the sibling `"persist"` span under a shared `"background snapshot"` root, and emit one span per compaction session rather than one per iteration. ### Why? The `compact()` function is called from the background persistence job, which runs inside a `"persisting background job"` span (inherited via `.in_current_span()` at task spawn time). Previously the `"compact database"` span was a child of that ambient span, making it appear nested under `"persisting background job"` in traces — inconsistent with its sibling spans `"snapshot"` and `"persist"`, which are already top-level root spans. Moving the span to the backend call site and giving it an explicit parent allows us to: 1. Detach it from the `"persisting background job"` ambient span. 2. Group it with the `"persist"` span under a shared `"background snapshot"` root span so all snapshot + compaction work appears together in trace viewers. 3. Keep `turbo-persistence` free of span-policy decisions (root vs. child), which belong at the call site where the ambient span context is known. ### How? - Introduced a non-entered `background_span` (`tracing::info_span!(parent: None, "background snapshot")`) as a shared root for both the snapshot/persist work and the compaction loop. - Passed `background_span.id()` as the `parent_span` argument to `snapshot_and_persist`, so the `"snapshot"` and `"persist"` spans become children of it. - Inside the idle compaction loop, a child `"compact database"` span is entered and dropped synchronously around each `compact()` call (not across the `tokio::select!` await point). - Moved the span creation outside the `MAX_IDLE_COMPACTION_PASSES` loop so there is one `"compact database"` span per compaction session rather than one per iteration. - Removed the `tracing::info_span!("compact database")` span from `turbo-persistence/src/db.rs` entirely. - Downgraded the `"sync new files"` span from `info` to `trace` level (noise reduction). The non-entered `tracing::Span` is `Send`, so it can be safely held across the `select!` await without triggering the `future cannot be sent between threads safely` compile error that an `EnteredSpan` would cause. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-20 15:03:22 +01:00
4 people	df886d4a2d	turbo-persistence: remove Unmergeable mmap advice (#91713) ... ### What? Remove the `MADV_UNMERGEABLE` (Unmergeable) mmap advice from `turbo-persistence`'s `advise_mmap_for_persistence` helper in `mmap_helper.rs`. ### Why? The WSL (Windows Subsystem for Linux) kernel does not support `MADV_UNMERGEABLE` and returns an error when it is applied, causing turbo-persistence to fail on WSL. See: https://x.com/von_cronen/status/2034944304712392891 Additionally, `MADV_UNMERGEABLE` is already the **default** state for memory pages — KSM (Kernel Same-page Merging) is opt-in via `MADV_MERGEABLE`. Explicitly setting `MADV_UNMERGEABLE` on pages that were never marked mergeable is a no-op on standard kernels, so the call provides no benefit while breaking WSL users. The `MADV_DONTFORK` advice is retained, as it has a clear correctness benefit (prevents mmap regions from being inherited by child processes on `fork()`). ### How? Removed the `mmap.advise(memmap2::Advice::Unmergeable)` call and its associated doc comment from `turbopack/crates/turbo-persistence/src/mmap_helper.rs`. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-20 14:59:04 +01:00
Tobias KoppersandGitHub	6a8a31a7f4	Fix layout segment optimization: move app-page imports to server-utility transition (#91701) ... ## Summary - Add `'turbopack-transition': 'next-server-utility'` to all server-side imports in the `app-page.ts` template so they are properly placed in the server utility layer, rather than relying on `SharedMerged` chunk groups to handle them - Remove the now-unnecessary `next-server-utility` transition from `fillMetadataSegment` import in `app_page_loader_tree.rs` - Add `SharedMultiple` chunk group variant to support multiple shared entries without a merge tag - This avoid passing the `parent` into `SharedMerged`, which did break the layout segment optimization. - Refactor `app_module_graphs` to combine `client_shared_entries` + `has_layout_segments` into a single `Option` parameter, simplifying the API and removing the dead code path for non-page endpoints ## Test plan - [ ] Verify dev and production builds work with app pages that have layout segments - [ ] Verify route handlers still build correctly (no client runtime entries passed) - [ ] Run existing app-dir e2e tests	2026-03-20 13:00:43 +01:00
4 people	046e0c4a00	turbopack-css: demote recoverable CSS parse warnings to Warning severity (#91524) ... ### What? Extends the CSS parse warning severity improvements in `turbopack/crates/turbopack-css/src/process.rs`. Canary (#90025, #91513) already: - Added a `severity` field to `ParsingIssue` and a `fn severity()` override - Demoted `UnsupportedPseudoClass` / `UnsupportedPseudoElement` selector errors from `Error` to `Warning` - Still **silently dropped** all other unrecognized parser warning kinds (`_ => { // Ignore }`) This PR completes the work: 1. **Surfaces all previously-ignored parser warnings as `IssueSeverity::Warning`** — the `_ => { // Ignore }` arm now emits a `ParsingIssue` at `Warning` severity instead of discarding it. 2. **Refactors the warning loop** — the nested severity match inside the existing `match err.kind` arm is replaced with a single top-level `let severity = match err.kind { … }` followed by an unconditional emit. This is cleaner and makes the severity logic easier to follow. Hard parse/transform errors (`Err(e)` from `parse_css_stylesheet`, minify errors, CSS module pure-selector errors) are unchanged — they remain `IssueSeverity::Error`. ### Why? Silently dropping parser warnings hides useful diagnostic information from developers. Surfacing them as `Warning` gives visibility without marking the build as broken. The restructure also avoids a redundant `UnsupportedPseudoClass`/`UnsupportedPseudoElement` check inside a nested match — the top-level arm order (specific pseudo variants before the catch-all `SelectorError(..)`) handles precedence correctly. ### How? - Removed the now-unused `error::SelectorError` short import (fully-qualified paths are used in the match). - Replaced the `match err.kind { <arms that emit> … _ => { // Ignore } }` pattern with: 1. `let severity = match err.kind { UnsupportedPseudo* => Warning, hard errors => Error, _ => Warning };` 2. Unconditional `IssueSource` construction and `ParsingIssue { severity, … }.emit()`. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-19 10:43:41 +00:00
4 people	0f089d5f9b	turbo-persistence: fix mmap page alignment and improve error context in MetaFile::open_internal (#91640) ... ### What? Two changes to `turbopack/crates/turbo-persistence/src/meta_file.rs` and `mmap_helper.rs`: 1. **Bug fix: mmap page alignment** — `MmapOptions::offset()` requires the offset to be page-aligned (typically 4096 bytes), but the code was passing `file.stream_position()` (the byte position after reading the variable-length header), which is not guaranteed to be page-aligned. This would cause `mmap` to fail or behave incorrectly on some inputs. 2. **Improved error context** — Add `.context("...")` to all `?` operators in `open_internal` (except read calls) and inside `advise_mmap_for_persistence`, so errors carry actionable context. The filename is already attached by the `MetaFile::open` wrapper, so no filename is repeated in these messages. ### Why? The mmap offset bug could cause silent failures or panics when the meta file header length is not a multiple of the OS page size. The missing error context made it hard to diagnose failures in production. ### How? **mmap fix (`meta_file.rs`):** - Remove `options.offset(offset)` — map the entire file from byte 0 instead. - Add a new `amqf_data_start: usize` field to `MetaFile` storing the header-end byte offset. - `amqf_data()` now returns `&self.mmap[self.amqf_data_start..]`, exposing only the AMQF data region. All existing callers that index into `amqf_data()` continue to work unchanged since their offsets are already relative to the start of AMQF data. **Error context (`meta_file.rs`):** - `File::open(path)?` → `.context("Failed to open meta file")` - `file.stream_position()?` → `.context("Failed to get stream position")` - `unsafe { options.map(&file) }.with_context(|| format!(...))` → `.context("Failed to mmap")` (drop the closure and filename, already provided by the outer wrapper) - `mmap.advise(Advice::Random)?` → `.context("Failed to advise mmap")` **Error context (`mmap_helper.rs`):** - `Advice::DontFork` → `.context("Failed to advise mmap DontFork")` - `Advice::Unmergeable` → `.context("Failed to advise mmap Unmergeable")` Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-19 10:21:46 +01:00
4 people	bdb2f2ce4d	Turbopack: Add importModule() support to webpack loaders (#89630) ... ## What? Adds support for `this.importModule()` in Turbopack's webpack loader compatibility layer. This API allows webpack loaders to dynamically import and execute modules (CJS, ESM, TypeScript, JSON, WebAssembly, etc.) during the build process, matching webpack's native `this.importModule()` behavior. ## Why? Webpack loaders like `@vanilla-extract/webpack-plugin`, `val-loader`, and custom loaders use `this.importModule()` to: - Load configuration files that are themselves modules with dependencies - Execute code at build time to generate derived source - Process dependency chains (e.g., a TS config that imports a CJS module that requires JSON) Without this, loaders relying on `importModule()` cannot work with Turbopack. ## How? ### Architecture The implementation follows a request/response pattern between the Node.js loader runner and the Rust-side Turbopack compiler: 1. **Loader calls `this.importModule(request)`** in the Node.js loader runner 2. **IPC message sent to Rust** with the module request and lookup path 3. **Rust resolves the module** using Turbopack's full resolver (with aliases, loader rules, etc.) 4. **Rust builds a Node.js bundle** containing the module and all its dependencies using Turbopack's code generation pipeline 5. **Bundle chunks sent back to Node.js** via IPC 6. **Node.js evaluates the bundle in-memory** using `vm.compileFunction` with a minimal CJS module system This approach leverages Turbopack's full module graph, so imported modules benefit from the same resolution (aliases, custom loaders, TypeScript support, etc.) as regular imports. ### Key Changes **Rust side (`turbopack-node/src/transforms/webpack.rs`):** - New `ImportModule` IPC response message type - Resolves modules through Turbopack's `esm_resolve` with full `AssetContext` - Generates a complete Node.js bundle (runtime + chunks + entry) for the imported module - Tracks file dependencies for proper cache invalidation **`SourceTransform` trait (`turbopack-core/src/source_transform.rs`):** - Added `asset_context` parameter to `SourceTransform::transform()` so transforms can access the full asset context for module resolution and bundling - Updated all implementors: `JsonSourceTransform`, `TextSourceTransform`, `BytesSourceTransform`, `MdxTransform`, `PostCssTransform`, `WebpackLoaderItems` **Node.js runtime (`turbopack-node/js/src/transforms/`):** - `webpack-loaders.ts`: Implements `this.importModule()` on the loader context, sends IPC request and evaluates the returned bundle - `webpack-loaders-runtime.ts`: In-memory CJS module evaluator that loads bundle chunks using `vm.compileFunction`, with support for: - Relative and absolute path resolution within the bundle - External package delegation to real Node.js `require()` - Patched `fs.createReadStream` for in-memory WebAssembly binary assets **New `ImportModule` reference subtype (`turbopack-core/src/reference_type.rs`):** - Added `EcmaScriptModulesReferenceSubType::ImportModule` to mark references created by importModule for proper handling in the module graph ### Test Coverage Comprehensive e2e test (`test/e2e/app-dir/webpack-loader-import-module/`) covering: - **TypeScript modules** with CJS and ESM dependencies - **CJS dependency chains** (TS → CJS → JSON) - **ESM `.mjs` modules** with shared dependencies - **Resolve aliases** (`alias-data` → `alias-data.mjs` via `resolveAlias` config) - **Custom loader rules** (`.custom-data` files processed by `text-to-export-loader`) - **Transitive dependencies** through aliases and custom loaders - **Turbopack-only features**: `new URL()` asset references, WebAssembly imports (`add.wasm`), and dynamic `import()` within importModule targets All tests pass for both webpack and Turbopack modes. --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Tobias Koppers <[email protected]>	2026-03-19 10:17:36 +01:00
4 people	f8f77da914	turbo-tasks-backend: batch schedule dirty tasks in aggregation_update (#91461) ... ### What? Batch the scheduling of dirty tasks during aggregation update processing instead of scheduling them one at a time. In `aggregation_update.rs`, `find_and_schedule_dirty_internal` previously called `ctx.schedule()` immediately for each task it decided to schedule. This involved a separate `ctx.task(id, TaskDataCategory::All)` call per task to read `leaf_distance` and `has_output` before calling `turbo_tasks.schedule(id, priority)`. ### Why? Each individual `ctx.task(...)` call may require restoring task data from persistent storage (disk I/O). When many tasks need scheduling — e.g. during a large invalidation wave, or initially when all session-dependent tasks are scheduled — doing this one-at-a-time is inefficient. `for_each_task_all` (via `prepare_tasks_with_callback`) can batch the restoration of persistent task data, amortizing I/O overhead across all tasks in the batch. ### How? - Added a `scheduled_tasks: FxHashMap<TaskId, TaskPriority>` field to `AggregationUpdateQueue` (skipped in bincode serialization — transient state that doesn't survive suspend/resume). - `find_and_schedule_dirty_internal` now inserts `(task_id, parent_priority)` into `scheduled_tasks` instead of calling `ctx.schedule()` directly. - At the end of `process()` (in the final `else` branch, when all sub-queues are exhausted), if `scheduled_tasks` is non-empty, `ctx.for_each_task_all` is called to batch-fetch the tasks and schedule them, then `scheduled_tasks` is cleared. This maximizes batch size by deferring all scheduling to the end of the operation. - `scheduled_tasks` is included in `is_empty()` (with exhaustive struct destructuring) so the compiler enforces the check stays up to date. --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-19 10:04:15 +01:00
4 people	6f669f1a73	Skip loadBindings() Lightning CSS check during next start (#91538) ... ### What? Skip the `loadBindings()` call that checks Lightning CSS support when `loadConfig()` is called during `PHASE_PRODUCTION_SERVER` (`next start`). ### Why? `loadConfig()` currently calls `loadBindings()` (which loads the native SWC `.node` binary) whenever `experimental.useLightningcss: true` is set, in order to check whether the binary supports Lightning CSS. This check gates a warning and a flag reset. During `next start`, the app is already fully built — no CSS compilation happens at serve time. Loading the native binary purely to run a capability check that has no effect is wasted work, and it delays server startup unnecessarily. ### How? - Promote `PHASE_PRODUCTION_SERVER` from a type-only import to a value import in `config.ts` so it can be used in a runtime comparison. - Add `phase !== PHASE_PRODUCTION_SERVER &&` as the first condition of the `if` block that calls `loadBindings()` for the Lightning CSS check. When the phase is `PHASE_PRODUCTION_SERVER`, the block is skipped entirely and `useLightningcss` is left as-is (unchanged). All other phases (`PHASE_DEVELOPMENT_SERVER`, `PHASE_PRODUCTION_BUILD`, `PHASE_EXPORT`, etc.) are unaffected — `loadBindings()` is still called and the capability check runs as before. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-19 09:00:42 +00:00
4 people	99d64f3ae1	[turbopack] Use bail! instead of panic! for duplicate module ident error (#91636) ... ### What? In `turbopack/crates/turbopack-core/src/module_graph/mod.rs`, the `#[cfg(debug_assertions)]` duplicate module ident detection block inside `SingleModuleGraph::new_inner` called `panic!` when duplicates were found. ### Why? `panic!` terminates the process hard, bypassing the normal error-handling chain. The enclosing function `new_inner` already returns `anyhow::Result<Vc<Self>>`, and `bail!` is already imported and used elsewhere in the same file. Using `bail!` converts this into a propagatable error, consistent with the rest of the error-handling idioms in the module. ### How? Single-line change: replace `panic!(...)` with `bail!(...)` at the duplicate module detection site. No new imports needed — `bail!` was already in scope via `use anyhow::{Context, Result, bail};`. Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-19 10:00:26 +01:00
4 people	6836c26b72	Restructure pr-status-triage skill: merge command into skill files (#91514) ... ### What? Migrates the `.claude/commands/pr-status.md` slash command into the `.agents/skills/pr-status-triage/` skill, restructuring the files so that `SKILL.md` has a single unified workflow and supporting details live in the appropriate reference files. Deletes the original command file. ### Why? The pr-status automated workflow existed in two places: a basic 6-step outline in `SKILL.md` and a detailed 10-step procedure in `.claude/commands/pr-status.md`. This duplication meant the skill was incomplete without the command file, and the command file couldn't benefit from the skill's supporting reference files (`workflow.md`, `local-repro.md`). ### How? **`SKILL.md`** — Replaced the old 6-step overview with a unified 7-step workflow covering: background `--wait` script launch, failure/review analysis, blocker-first prioritization, flaky test checking, local reproduction, resolving review threads, and retriggering CI when only flaky tests remain. Step 7 now includes a retry loop: after retriggering, wait 5 minutes and re-triage from step 1, up to 5 times. **`workflow.md`** — Extended with content from the command file: subagent prompt templates (CI failures + review comments), summary table format examples, failure category taxonomy (8 categories), and review comment category taxonomy (7 types). Existing sections (prioritization order, failure handling rules, common patterns, resolving review threads) preserved. **`local-repro.md`** — Expanded the "Match CI Environment Variables" section with feature flag details (`__NEXT_USE_NODE_STREAMS`, `__NEXT_CACHE_COMPONENTS`), the `NEXT_SKIP_ISOLATE` warning for module resolution fixes, and a concrete example command. **`AGENTS.md`** — Updated the stale `/pr-status` command reference to point to the `$pr-status-triage` skill. **`.claude/commands/pr-status.md`** — Deleted (all content migrated). ### Files changed | File | Action | |------|--------| | `.agents/skills/pr-status-triage/SKILL.md` | Rewritten with unified workflow (includes retry loop for flaky CI) | | `.agents/skills/pr-status-triage/workflow.md` | Extended with templates, tables, categories | | `.agents/skills/pr-status-triage/local-repro.md` | Expanded env var section | | `AGENTS.md` | Updated skill reference (1 line) | | `.claude/commands/pr-status.md` | Deleted | --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>	2026-03-19 10:00:10 +01:00