mirror of
https://github.com/hanzoai/next.js.git
synced 2026-10-30 14:46:38 +00:00
8db9a75243
### What? Re-lands #91576 ("turbo-tasks: add hashed cell mode for hash-based change detection without cell data"), which was reverted in #92103 due to a `FATAL` crash in the `filesystem-cache` test suite. Includes a bug fix on top: in `task_execution_completed_prepare`, skip updating `cell_type_max_index` when the task completed with an error. Also adds a `CellHash = [u8; 16]` type alias (requested in review) used throughout the hash pipeline. ### Why? **The original feature** (`serialization = "hash"` on `FileContent` and `Code`) stores a hash of the cell data instead of the full serialized value. On session restore, the hash is used to detect whether cell content has changed without needing the full data in memory. This avoids a large persistent cache size increase. **The bug** that caused the revert: When a task fails partway through re-execution (before recreating all the cells from its previous run), `cell_counters` only reflects the partially-executed state. The old code used those partial counters to update `cell_type_max_index`, removing entries for cell types that were not yet created at the point of failure. This caused downstream tasks that still held cell dependencies from the previous successful run to hit a hard "Cell no longer exists" error. **Concrete failure path** in `filesystem-cache rename app page` test: 1. `get_app_page_entry` runs for `/remove-me/page`, creating two `FileContent` cells (indices 0 and 1). `cell_type_max_index[FileContent] = 2` is persisted. 2. The folder is renamed (`app/remove-me` → `app/add-me`), dirtying the task. 3. On re-execution, `get_app_page_entry` fails at `config.await?` (the loader tree errors because the directory is gone) — before any `FileContent::cell()` calls. 4. `cell_counters` has no `FileContent` entry → old code removed `cell_type_max_index[FileContent]`. 5. The `parse` task tries to read `FileContent` cell 1 from `get_app_page_entry` → `cell_type_max_index` is `None` → **"Cell no longer exists" panic → FATAL error**. **Why it didn't crash before** `serialization = "hash"`: `FileContent` was previously serializable, so `parse` read stale cell data directly from `persistent_cell_data`, which `task_execution_completed_cleanup` already preserves on error. With `serialization = "hash"`, data is transient — readers fall back to `cell_type_max_index` for range validation, where a stale `None` caused the crash. ### How? #### Core feature: `serialization = "hash"` cell mode - New `SerializationMode::Hash` variant in `turbo-tasks-macros` — marks a value type as non-serializable but stores a `DeterministicHash` of the cell data for change detection. - `VcCellHashedCompareMode<T>` cell mode: compares values via `PartialEq` when available, falls back to hash comparison when transient data has been evicted. - `hashed_compare_and_update` / `hashed_compare_and_update_with_shared_reference` on `CurrentCellRef` compute and pass content hashes through the update pipeline. - Backend `update_cell` uses hash-based comparison to skip invalidation when the old cell data is unavailable but the hash matches. - `cell_data_hash: AutoMap<CellId, CellHash>` field in task storage persists hashes across sessions. - Stale `cell_data_hash` entries are cleaned up in `task_execution_completed_cleanup` alongside cell data removal. - `CellHash = [u8; 16]` type alias keeps alignment at 1 byte to avoid padding growth in `AutoMap`/`LazyField` enum variants. - Hash bytes use little-endian encoding (`to_le_bytes`) for cross-platform cache portability. #### Bug fix: preserve `cell_type_max_index` on task error In `task_execution_completed_prepare`, guard the `cell_type_max_index` update block with `if result.is_ok()`. This mirrors the existing `task_execution_completed_cleanup` behavior that already skips cell data removal when `is_error` is true, keeping `cell_type_max_index` consistent with the preserved transient cell data. #### Applied to `FileContent` and `Code` - `FileContent` uses `serialization = "hash"` — full content is persisted via a separate `PersistedFileContent` type when needed (e.g., in `DiskFileSystem::write`). - `Code` uses `serialization = "hash"` with `Arc<Vec<Mapping>>` for cheap cloning. `Code::cell_persisted()` creates a `PersistedCode` cell directly and returns `Vc<Code>` via `PersistedCode::to_code()`, avoiding an intermediate hash-mode cell. #### Other improvements - `DeterministicHash` impls for `SmallVec` and `()`. - `Xxh3Hash128Hasher::finish_bytes()` method returning `[u8; 16]`. - `hash = "manual"` option on `#[turbo_tasks::value]` to opt out of auto-deriving `DeterministicHash`. **Note:** The shutdown hang and cache poisoning fixes that were previously on this branch have been merged separately via #92254. ### Test plan - [x] `test/e2e/filesystem-cache/filesystem-cache.test.ts` passes (all 17 tests) - [x] New `turbopack/crates/turbo-tasks-backend/tests/hashed_cell_mode.rs` integration test verifies hash-based change detection: value changes trigger invalidation, equal values (same hash) do not - [x] `cargo check` passes for `turbo-tasks`, `turbo-tasks-backend`, `turbo-tasks-fs`, `turbopack-core`, `turbopack-ecmascript` - [x] CI green (attempt 2) <!-- NEXT_JS_LLM_PR --> --------- Co-authored-by: Tobias Koppers <[email protected]> Co-authored-by: Claude <[email protected]>
22 lines
395 B
TOML
22 lines
395 B
TOML
[package]
|
|
name = "turbo-tasks-hash"
|
|
version = "0.1.0"
|
|
description = "TBD"
|
|
license = "MIT"
|
|
edition = "2024"
|
|
autobenches = false
|
|
|
|
[lib]
|
|
bench = false
|
|
|
|
[lints]
|
|
workspace = true
|
|
|
|
[dependencies]
|
|
bincode = { workspace = true }
|
|
data-encoding = { workspace = true }
|
|
smallvec = { workspace = true }
|
|
sha2 = { workspace = true }
|
|
turbo-tasks-macros = { workspace = true }
|
|
xxhash-rust = { workspace = true }
|