* SEC-10179 Integrate test system IO for Playwright and Cypress (#36376) * change retry to 1, fixed and disabled failed tests * add v2 templates for Cypress and Playwright E2E tests with test system io integration * add commenting to pr * identify more playwrights to fix separately * disable deletion-report.spec for separate fix --------- Co-authored-by: Mattermost Build <[email protected]> * fix: permission required by test-system-io actions (#36477) * fix: permission required by e2e test-system-io actions (#36478) Co-authored-by: sabril <[email protected]> * chore(ci): consolidate openldap runner prep into a composite action (#36563) Co-authored-by: sabril <[email protected]> * ci: port test-system-io v2 templates from release-11.9 Bring Playwright/Cypress template-v2 in line with release-11.9 so they include prepare-run/prep-deps caching, runner-prep-openldap, and pinned mattermost-test-system-io action SHAs (1631d8fcea24). Bump Playwright v2 workers to 10 to match 11.8/11.9. Prerequisite for cherry-picking #37570 (testcontainers) onto release-11.7. Co-authored-by: sabril <[email protected]> * E2E/Playwright: Add testcontainers to playwright-lib (#37570) * add testcontainers to playwright-lib * include retry mechanism for transient failures * - Introduced a new command `testcontainers:up` in package.json to run Playwright tests with Testcontainers. - Created a standalone Playwright configuration file `playwright.testcontainers-up.config.ts` for managing Testcontainers. - Added a no-op test `testcontainers_up.spec.ts` to ensure the Testcontainers stack is up during the test run. - Implemented a global setup script `testcontainers_up_global_setup.ts` to start and stop the Testcontainers stack. - Updated dependencies in package.json, including adding `chalk` for logging. * fix package-lock * fix tsc and restore waiting for all migrations to complete --------- Co-authored-by: Mattermost Build <[email protected]> * fix(e2e): remove obsolete CRT intro modal dismiss from uiClickSidebarItem The CRT tutorial modal was removed in MM-66470; dismissing #genericModalLabel races and flakes MM-T4261_3. Co-authored-by: sabril <[email protected]> * fix(e2e): sync Playwright navigations to live testcontainers baseURL After restartMattermostContainer remaps the host port, relative page.goto must follow testConfig.baseURL rather than the worker-start config value. Co-authored-by: sabril <[email protected]> * fix(e2e): restore eslint-disable for axe empty fixture pattern Co-authored-by: sabril <[email protected]> * fix(e2e): ignore webhook assets in eslint; prettier README Co-authored-by: sabril <[email protected]> * fix(e2e): prettier-format testcontainers config helpers Co-authored-by: sabril <[email protected]> * ci(e2e): fully adopt test-system-io and lighten Playwright services Port #37413: drop the dual v1/v2 dispatch path, promote the test-system-io templates to the canonical names, and remove legacy calculate-results / AWS artifact upload plumbing. For Playwright testcontainers on this release, leave optional sidecars (openldap, keycloak, elasticsearch, opensearch, minio, azurite, webhook) off by default and remove the related service-specific specs that are not part of the older suite. Co-authored-by: sabril <[email protected]> * style(e2e): prettier stack.ts after optional webhook change Co-authored-by: sabril <[email protected]> * ci(e2e): grant contents:read to test-system-io workflow callers Reusable Playwright/Cypress templates require contents:read. Without it on the caller job, GitHub Actions fails the workflow at startup. Co-authored-by: sabril <[email protected]> * ci: retrigger enterprise build after transient npm cache failure Co-authored-by: sabril <[email protected]> * test(e2e): align testcontainers feature flags with release docker-compose Mirror MM_FEATUREFLAGS_* from e2e-tests/.ci/server.generate.sh for each release, dropping flags that do not exist in that release's FeatureFlags struct (or were never enabled in the compose stack). Co-authored-by: sabril <[email protected]> * ci: retrigger enterprise after flaky message_export Postgres failure Co-authored-by: sabril <[email protected]> * e2e(playwright): omit unset Testcontainers env keys; align release docs When optional webhook/sidecars are off, .env.testcontainers no longer writes PW_*=undefined. Docs match empty DEFAULT_TESTCONTAINERS_SERVICES. Co-authored-by: sabril <[email protected]> * e2e(playwright): prettier-format lib/README.md table Fixes CI npm run check (prettier) failures from the services table edit. Co-authored-by: sabril <[email protected]> * ci: retrigger E2E after Cypress commit-status race Orchestration reported 1 failed unit before retest; final summary was 100% but commit status stayed failure. Retrigger for a clean green. Co-authored-by: sabril <[email protected]> * e2e(cypress): keep tests/support/env import on release-11.7 #36376 switched message_deletion_spec to @/support/env, but this release has no Cypress webpack @ alias (added later). That caused a Webpack Module not found failure and a stuck failed commit status. Co-authored-by: sabril <[email protected]> --------- Co-authored-by: Mattermost Build <[email protected]> Co-authored-by: sabril <[email protected]> Co-authored-by: yasser khan <[email protected]> Co-authored-by: Hanzo Dev <[email protected]>
@mattermost/playwright-lib
A comprehensive end-to-end testing library for Mattermost web, desktop and plugin applications using Playwright.
Overview
This library provides:
- Pre-built page objects and components for common Mattermost UI elements
- Server configuration and initialization utilities
- Test fixtures and helpers
- Visual testing support with Percy integration
- Accessibility testing support with axe-core
- Browser notification mocking
- File handling utilities
- Common test actions and assertions
Installation
npm install @mattermost/playwright-lib
Peer Dependencies
This library requires @playwright/test version 1.55.0 or higher:
npm install @playwright/test@^1.55.0
Usage
Basic example of logging in and posting a message:
import {test, expect} from '@mattermost/playwright-lib';
test('user can post message', async ({pw}) => {
// # Create and login a new user
const {user} = await pw.initSetup();
const {channelsPage} = await pw.testBrowser.login(user);
// # Navigate and post a message
await channelsPage.goto();
const message = 'Hello World!';
await channelsPage.postMessage(message);
// * Verify message appears
const lastPost = await channelsPage.getLastPost();
await expect(lastPost).toHaveText(message);
});
Key Components
Page Objects
Ready-to-use page objects for common Mattermost pages:
- Login
- Signup
- Channels
- System Console
- And more...
UI Components
Reusable component objects for UI elements:
- Headers
- Posts
- Menus
- Modals
- And more...
Test Utilities
Helper functions for common testing needs:
- Server setup and configuration
- User/team creation
- File handling
- Visual testing
- And more...
Configuration
The library can be configured via optional environment variables:
Environment Variables
All environment variables are optional with sensible defaults.
Server Configuration
| Variable | Description | Default |
|---|---|---|
PW_BASE_URL |
Server URL | http://localhost:8065 |
PW_ADMIN_USERNAME |
Admin username | sysadmin |
PW_ADMIN_PASSWORD |
Admin password | Sys@dmin-sample1 |
PW_ADMIN_EMAIL |
Admin email | [email protected] |
PW_ENSURE_PLUGINS_INSTALLED |
Comma-separated list of plugins to install | [] |
PW_RESET_BEFORE_TEST |
Reset server before test | false |
High Availability Cluster Settings
| Variable | Description | Default |
|---|---|---|
PW_HA_CLUSTER_ENABLED |
Enable HA cluster | false |
PW_HA_CLUSTER_NODE_COUNT |
Number of cluster nodes | 2 |
PW_HA_CLUSTER_NAME |
Cluster name | mm_dev_cluster |
Push Notifications
| Variable | Description | Default |
|---|---|---|
PW_PUSH_NOTIFICATION_SERVER |
Push notification server URL | https://push-test.mattermost.com |
Playwright Settings
| Variable | Description | Default |
|---|---|---|
PW_HEADLESS |
Run tests headless | true |
PW_SLOWMO |
Add delay between actions in ms | 0 |
PW_WORKERS |
Number of parallel workers | 1 |
Visual Testing
| Variable | Description | Default |
|---|---|---|
PW_SNAPSHOT_ENABLE |
Enable snapshot testing | false |
PW_PERCY_ENABLE |
Enable Percy visual testing | false |
CI Settings
| Variable | Description | Default |
|---|---|---|
CI |
Set automatically in CI environments | N/A |
Testcontainers
Selects testcontainers mode — Playwright brings up the server + dependencies itself via Testcontainers — instead of the external mode default, which expects a server already running at PW_BASE_URL.
| Variable | Description | Default |
|---|---|---|
PW_USE_TESTCONTAINERS |
Selects testcontainers mode |
false |
PW_TESTCONTAINERS_SERVICES |
Comma-separated additional services to start (openldap, keycloak, elasticsearch, opensearch, minio, azurite); unset/empty starts none (core: Postgres, Inbucket, Mattermost) |
none (empty) |
PW_TESTCONTAINERS_WEBHOOK |
Start the webhook sidecar (true to enable) |
false |
PW_TESTCONTAINERS_REUSE |
Reuse containers across repeated local runs instead of recreating them; tear down with npm run testcontainers:down (requires testcontainers.reuse.enable=true in ~/.testcontainers.properties) |
true |
PW_TESTCONTAINERS_CONTAINER_RUNNER |
Set when the Playwright process itself runs inside a Docker container (e.g. CI), to join it to the Testcontainers network | false |
SERVER_IMAGE |
Prebuilt Mattermost server image testcontainers mode starts |
mattermostdevelopment/mattermost-enterprise-edition:master |
MM_ENV |
Comma-separated KEY=VALUE server config overrides, merged over the test baseline |
none |
PW_LDAP_HOST / PW_LDAP_PORT |
OpenLDAP host/port (only used when openldap is started) |
localhost / 389 |
PW_KEYCLOAK_URL |
Keycloak URL (only used when keycloak is started) |
http://localhost:8484 |
PW_ELASTICSEARCH_URL |
Elasticsearch URL (only used when elasticsearch is started) |
http://localhost:9200 |
PW_OPENSEARCH_URL |
OpenSearch URL (only used when opensearch is started) |
http://localhost:9201 |
PW_MINIO_URL |
Minio URL (only used when minio is started) |
http://localhost:9000 |
PW_AZURITE_URL |
Azurite URL (only used when azurite is started) |
http://localhost:10000 |
In testcontainers mode, these host/port defaults are never actually used — Testcontainers always assigns its own dynamic port or network alias per run, so a testcontainers-mode run can never collide with a same-machine external-mode session using the fixed defaults above. Tear a reused stack down explicitly with npm run testcontainers:down (from the playwright/ package).
Accessibility Testing
The library includes built-in accessibility testing using axe-core:
import {test, expect} from '@mattermost/playwright-lib';
test('verify login page accessibility', async ({page, axe}) => {
// # Navigate to login page
await page.goto('/login');
// # Run accessibility scan
const results = await axe.builder(page).analyze();
// * Verify no accessibility violations
expect(results.violations).toHaveLength(0);
});
The axe-core integration:
- Runs WCAG 2.0 Level A & AA rules by default
- Provides detailed violation reports
- Supports rule customization
- Can be configured per-test or globally
Visual Testing
The library supports visual testing through Playwright's built-in visual comparisons and Percy integration:
import {test, expect} from '@mattermost/playwright-lib';
test('verify channel header appearance', async ({pw, browserName, viewport}, testInfo) => {
// # Setup and login
const {user} = await pw.initSetup();
const {page, channelsPage} = await pw.testBrowser.login(user);
// # Navigate and prepare page
await channelsPage.goto();
await expect(channelsPage.appBar.playbooksIcon).toBeVisible();
await pw.hideDynamicChannelsContent(page);
// * Take and verify snapshot
await pw.matchSnapshot(testInfo, {page, browserName, viewport});
});
Browser Notifications
Mock and verify browser notifications:
import {test, expect} from '@mattermost/playwright-lib';
test('verify notification on mention', async ({pw}) => {
// # Setup users and team
const {team, adminUser, user} = await pw.initSetup();
// # Setup admin browser with notifications
const {page: adminPage, channelsPage: adminChannelsPage} = await pw.testBrowser.login(adminUser);
await adminChannelsPage.goto(team.name, 'town-square');
await pw.stubNotification(adminPage, 'granted');
// # Setup user browser and post mention
const {channelsPage: userChannelsPage} = await pw.testBrowser.login(user);
await userChannelsPage.goto(team.name, 'off-topic');
await userChannelsPage.postMessage(`@ALL good morning, ${team.name}!`);
// * Verify notification received
const notifications = await pw.waitForNotification(adminPage);
expect(notifications.length).toBe(1);
});
Contributing
See CONTRIBUTING.md for development setup and guidelines.
License
See LICENSE.txt for license information.