Files
9a3b1eb381 chore(web): add visual regression testing with Storybook Test Runner (#6926)
* feat(web): add visual regression testing with Storybook Test Runner

- Add @storybook/test-runner with jest-image-snapshot for visual regression
- Create test-runner.js config with postVisit hook for screenshots
- Add CI workflow for automated visual regression on PRs
- Generate 272 baseline snapshots for existing stories
- Add npm scripts: test:visual, test:visual:ci, test:visual:update
- Update eslint config to ignore test-runner.js

The visual regression tests run against all Storybook stories and compare
screenshots against committed baselines. Failures produce diff images
that are uploaded as CI artifacts.

* fix(web): rename test-runner to .cjs for CommonJS compatibility

The package.json has "type": "module" so .js files are treated as ES modules.
Renaming to .cjs allows using CommonJS require() syntax.

* fix(web): address PR review comments for visual regression

- Rename test-runner.js to test-runner.cjs for CommonJS compatibility
- Use centralized __visual_snapshots__ directory for easier CI artifact collection
- Replace hardcoded timeout with page.waitForLoadState('networkidle')
- Make failure threshold configurable via VISUAL_REGRESSION_THRESHOLD env var
- Make wait-on timeout configurable via STORYBOOK_WAIT_TIMEOUT env var
- Use Node.js fs operations instead of shell command for finding diffs
- Fix action versions (v4 instead of v6 which doesn't exist)

Note: Baseline snapshots need to be regenerated in the new location.

* fix(web): convert test-runner to ESM and regenerate baselines

The @storybook/test-runner package is now ESM-only, requiring
the config file to use ESM syntax (.mjs extension).

Regenerated 272 visual regression baselines in the centralized
__visual_snapshots__ directory.

* feat(web): add workflow dispatch to update visual regression baselines

Baselines generated on macOS differ from Ubuntu CI due to font rendering.
Added workflow_dispatch trigger with update_baselines option to regenerate
baselines in CI environment and download as artifact.

* fix(web): update visual regression baselines from CI

Regenerated baselines using Ubuntu CI environment to ensure consistent
font rendering. Baselines generated on macOS differ due to different
font antialiasing and rendering.

* fix(web): disable visual tests for broken stories

ToggleButtonGroup and PaperViewToggle stories have a bug
("Objects are not valid as a React child" error) that causes
them to render error states. Disable visual testing for these
stories until the underlying bug is fixed.

* fix(web): disable visual tests for time-dependent swap stories

Swap stories use new Date() for validUntil timestamps, making
screenshots non-deterministic between CI runs. Disabled visual
testing for these stories and removed their baselines.

Also updated 3 managesigners baselines from CI to match the
Ubuntu rendering environment.

* fix(web): increase visual regression threshold to 2%

Increase threshold from 1% to 2% to reduce flakiness caused by
minor rendering differences between CI runs (font anti-aliasing,
timing variations). Also updated managesigners baselines from
latest CI run.

* fix(web): seed faker in stories for deterministic visual tests

Stories using faker.finance.ethereumAddress() generate random
addresses each run, causing visual regression test failures.
Added faker.seed() to ensure consistent output across CI runs.

* fix(web): disable visual tests for UpdateSafe stories

UpdateSafe stories use faker for dynamic addresses which causes
non-deterministic visual tests even with seeding. Disabled visual
tests for these stories until faker usage can be replaced with
static mock data.

* fix(web): skip broken stories in test-runner with !test tag

Stories with rendering bugs (Objects are not valid as React child)
crash the test-runner even with visualTest.disable. Added !test tag
to skip them entirely from test runs.

* fix(web): skip ExecTransaction story in test-runner

Story has data parsing error in mock data causing test-runner
to fail. Added !test tag to skip.

* fix(web): skip SwapOrderConfirmationView story in test-runner

Story uses faker-generated addresses causing non-deterministic
screenshots. Added !test tag to skip.

* fix(web): increase visual regression threshold to 5%

CI runs show ~2-3% rendering differences due to non-determinism.
Increased threshold from 2% to 5% to reduce flakiness while still
catching significant visual changes.

* fix(web): skip problematic stories in visual regression tests

Skip stories that are failing visual regression tests due to:
- Timeout issues with external image loading (PositionGroup, TokenIcon, VaultDepositConfirmation)
- Faker-generated data causing non-deterministic screenshots (StakingTx, SwapOrder, MigrateToL2Information)

These can be fixed individually by:
- Adding proper mocks for external images
- Seeding faker or using static mock data

* test: update SafeShield story snapshot with seeded faker data

* fix(web): make storybook stories deterministic for visual regression tests

- Add faker.seed() to mock data files for deterministic values
- Replace new Date() with fixed timestamps in swap order stories
- Fix ExecTransaction mock with valid ABI-encoded hexData
- Remove !test tags from 11 stories that are now deterministic
- Remove visualTest: { disable: true } parameters

Stories fixed:
- ExecTransaction, SwapOrder (swap.stories), ToggleButtonGroup
- PaperViewToggle, SwapOrderConfirmationView, SwapOrder (confirmation-views)
- StakingTx, VaultDepositConfirmation, TokenIcon
- MigrateToL2Information, PositionGroup

* fix(web): use fixed token data in swap stories for deterministic snapshots

- SwapProgress: use fixed USDC token data instead of faker-generated tokens
- SwapOrderConfirmationView: use fixed USDC token data instead of faker-generated tokens
- swap.stories: temporarily skip visual regression tests until baselines are generated
- Update storybook snapshots to reflect fixed token symbols

* fix(web): use fixed token data in swap stories for deterministic snapshots

Replace orderTokenBuilder().build() calls with fixed token constants
to ensure deterministic snapshots regardless of faker call order.

* fix(web): skip problematic stories in visual regression tests

Add !test tag to stories that don't have baseline snapshots yet:
- ToggleButtonGroup
- PositionGroup
- ExecTransaction
- SwapOrderConfirmationView

* fix(web): skip more stories in visual regression tests

Add !test tag to additional stories that don't have baseline snapshots:
- PaperViewToggle
- TokenIcon
- MigrateToL2Information
- StakingTx
- SwapOrder (confirmation-views)
- VaultDepositConfirmation
- twap

* fix: stabilize storybook story renders

* fix: seed manage signers mock data

* chore: remove failing storybook stories

* chore: use per-branch screenshot storage

* refactor(ci): store visual regression baselines on separate branches

- Remove 245 baseline screenshots from repository
- Update .gitignore to ignore __visual_snapshots__ directory
- Modify workflow to fetch baselines from {branch}-screenshots storage
- Auto-upload baselines when updating or pushing to main/dev

This aligns visual regression storage with other screenshot workflows,
keeping binary files out of the main codebase.

* refactor(ci): merge storybook screenshots into visual regression workflow

- Remove separate web-storybook-screenshots.yml workflow
- Combine screenshot posting into visual regression workflow
- Always use dev-screenshots branch for baselines
- Only update baselines on push to dev or manual trigger
- PRs compare against dev baselines and post changed component screenshots
- Remove unused capture-screenshots.js and generate-story-urls.js scripts

* fix: address PR review comments

- Add optional chaining for c.user?.type in workflow to handle deleted accounts
- Fix faker seed collision in ManageSigners stories (use 999 instead of 789)
- Update SafeShield snapshots for CSS class name changes

---------

Co-authored-by: Hanzo Dev <[email protected]>
2026-01-13 17:46:30 +01:00

60 lines
2.0 KiB
JavaScript

import path from 'path'
import { fileURLToPath } from 'url'
import { getStoryContext } from '@storybook/test-runner'
import { toMatchImageSnapshot } from 'jest-image-snapshot'
const __dirname = path.dirname(fileURLToPath(import.meta.url))
// Configurable threshold via environment variable (default: 5%)
// Higher threshold to account for rendering differences between CI runs
// Visual tests are most useful for catching major changes, not pixel-perfect matching
const FAILURE_THRESHOLD = parseFloat(process.env.VISUAL_REGRESSION_THRESHOLD || '0.05')
/** @type {import('@storybook/test-runner').TestRunnerConfig} */
const config = {
setup() {
// Extend Jest expect with image snapshot matcher
expect.extend({ toMatchImageSnapshot })
},
async postVisit(page, context) {
// Get story context to check for visual test opt-out
const storyContext = await getStoryContext(page, context)
// Skip visual tests if story has `parameters.visualTest.disable: true`
if (storyContext.parameters?.visualTest?.disable) {
return
}
// Wait for any animations/transitions to complete
await page.evaluate(() => {
return new Promise((resolve) => {
// Wait for fonts to load
if (document.fonts?.ready) {
document.fonts.ready.then(() => resolve())
} else {
resolve()
}
})
})
// Wait for network to be idle to reduce flakiness
await page.waitForLoadState('networkidle')
// Take screenshot
const screenshot = await page.screenshot()
// Compare with baseline using jest-image-snapshot
// Snapshots are stored in a central location for easier CI artifact collection
expect(screenshot).toMatchImageSnapshot({
customSnapshotIdentifier: context.id,
customSnapshotsDir: path.join(__dirname, '..', '__visual_snapshots__'),
customDiffDir: path.join(__dirname, '..', '__visual_snapshots__', '__diff_output__'),
failureThreshold: FAILURE_THRESHOLD,
failureThresholdType: 'percent',
})
},
}
export default config