# E2E Test Pipelines Three automated E2E test pipelines cover different stages of the development lifecycle. ## Pipelines | Pipeline | Trigger | Editions Tested | Image Source | |----------|---------|----------------|--------------| | **PR** (`e2e-tests-ci.yml`) | Argo Events on `Enterprise CI/docker-image` status | enterprise | `mattermostdevelopment/**` | | **Merge to master/release** (`e2e-tests-on-merge.yml`) | Platform delivery after docker build (`delivery-platform/.github/workflows/mattermost-platform-delivery.yaml`) | enterprise, fips | `mattermostdevelopment/**` | | **Release cut** (`e2e-tests-on-release.yml`) | Platform release after docker build (`delivery-platform/.github/workflows/release-mattermost-platform.yml`) | enterprise, fips, team (future) | `mattermost/**` | All pipelines follow the **smoke-then-full** pattern: smoke tests run first, full tests only run if smoke passes. ## Workflow Files ``` .github/workflows/ ├── e2e-tests-ci.yml # PR orchestrator ├── e2e-tests-on-merge.yml # Merge orchestrator (master/release branches) ├── e2e-tests-on-release.yml # Release cut orchestrator ├── e2e-tests-cypress.yml # Shared wrapper: routes to v1 or v2 template ├── e2e-tests-playwright.yml # Shared wrapper: routes to v1 or v2 template ├── e2e-tests-cypress-template-v2.yml # Active: cypress + test-system-io dispatch ├── e2e-tests-playwright-template-v2.yml # Active: playwright + test-system-io dispatch ├── e2e-tests-cypress-template.yml # Deprecated v1 (legacy in-job execution) └── e2e-tests-playwright-template.yml # Deprecated v1 (legacy in-job execution) ``` > **v1 templates are deprecated.** They remain available behind a feature flag during cutover but receive no further changes. New work targets the v2 templates exclusively. The wrappers route by `vars.E2E_USE_TEST_IO_DISPATCH` — `'true'` selects v2, anything else falls back to v1. ### Call hierarchy ``` e2e-tests-ci.yml ─────────────────┐ e2e-tests-on-merge.yml ───────────┤──► e2e-tests-cypress.yml ─────┐ e2e-tests-on-release.yml ─────────┘ e2e-tests-playwright.yml ──┤ │ ┌──────────────────────────┘ │ routes on E2E_USE_TEST_IO_DISPATCH ▼ v2 (active) ──► e2e-tests-{cypress,playwright}-template-v2.yml v1 (legacy) ──► e2e-tests-{cypress,playwright}-template.yml ``` --- ## Workflow Architecture (v2) v2 splits the template into five jobs — `prepare-run`, `prep-deps`, `dispatch-begin`, `workers` (matrix), and `report` — and pushes spec-level execution to [Test System IO](https://github.com/mattermost/mattermost-test-system-io) so workers stay thin and identical. ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Template v2: e2e-tests-{cypress,playwright}-template-v2.yml │ │ │ │ ┌───────────────────┐ ┌──────────────────────────────┐ │ │ │ prepare-run │ │ prep-deps │ │ │ │ (1 runner) │ parallel │ (1 runner) │ │ │ │ │ ◄────────────► │ │ │ │ │ • build workers │ │ Cypress: │ │ │ │ matrix [1..N] │ │ • cypress/node_modules │ │ │ │ • compute commit │ │ • ~/.cache/Cypress (binary)│ │ │ │ status context │ │ │ │ │ │ • emit composite │ │ Playwright: │ │ │ │ identity │ │ • webapp/platform/{client, │ │ │ │ │ │ types}/{lib,node_mod} │ │ │ │ │ │ • playwright/node_modules │ │ │ │ │ │ • playwright/lib/dist │ │ │ │ │ │ • ~/.cache/ms-playwright │ │ │ │ │ │ (chromium only) │ │ │ │ │ │ │ │ │ │ │ │ → saved to actions/cache │ │ │ └─────────┬─────────┘ └───────────────┬──────────────┘ │ │ │ │ │ │ │ ▼ │ │ │ ┌──────────────────────────────┐ │ │ │ │ dispatch-begin │ │ │ │ │ • register run with │ │ │ │ │ Test System IO │ │ │ │ │ • runs immediately before │ │ │ │ │ workers to minimise the │ │ │ │ │ inactivity-timeout window │ │ │ │ └───────────────┬──────────────┘ │ │ │ │ │ │ └────────────────────┬─────────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ workers (matrix, fail-fast: false) │ │ │ │ Cypress full: N=40 | Playwright full: N=10 │ │ │ │ │ │ │ │ each worker, in parallel: │ │ │ │ 1. sparse-checkout actions + full checkout-repo │ │ │ │ 2. setup-node │ │ │ │ 3. restore caches ◄─── actions/cache (from prep-deps) │ │ │ │ (fail-on-cache-miss: true) │ │ │ │ 4. cloud-init + start-server (docker compose stack) │ │ │ │ 5. prepare-cypress | prepare-playwright (run setup project) │ │ │ │ 6. dispatch-run ──────────────────────────────────┐ │ │ │ │ (pulls specs from Test System IO, runs locally, │ │ │ │ │ posts result, loops until queue is empty) │ │ │ │ │ 7. cloud-teardown │ │ │ │ └────────────────────┬────────────────────────────────────┼───────┘ │ │ │ │ │ │ ▼ │ │ │ ┌─────────────────────────────────────────────────┐ │ │ │ │ report │ │ │ │ │ • pull aggregated results from Test System IO │ ◄────┘ │ │ │ • post commit status │ │ │ │ • send webhook notification │ │ │ └─────────────────────────────────────────────────┘ │ └────────────────────────────────────────────────────────────────────────┬─┘ │ ┌───────────────────────────▼───┐ │ Test System IO (external) │ │ • spec-level dispatch │ │ • result aggregation │ │ • retry orchestration │ └───────────────────────────────┘ ``` ### Key properties - **Spec-level vs. job-level parallelism.** The matrix sizes the runner pool; Test System IO does the spec assignment. Slow specs don't block a worker — fast workers keep pulling the next spec from the queue. - **Cache-only workers.** `prep-deps` installs once per workflow run and saves to `actions/cache`. Every worker restores with `fail-on-cache-miss: true` and runs zero `npm ci`. Eliminates the 40-way `EEXIST/ENOENT` race in npm's shared cacache writer. - **dispatch-begin runs late.** It depends on `prep-deps` so the gap between Test System IO run registration and the first worker calling `dispatch-run` is just per-worker setup (~3–5 min). Registering earlier risks the run timing out before any worker checks in, bulk-failing every spec. - **Playwright slim slice.** Playwright only consumes `@mattermost/client` and `@mattermost/types` from webapp, so prep-deps caches just those two packages' built `lib/` and `node_modules` (~10–30 MB) instead of the full `webapp/node_modules` tree (~1–2 GB). - **Browser/binary caches.** Cypress caches `~/.cache/Cypress` (cypress binary lives outside node_modules); playwright caches `~/.cache/ms-playwright` (chromium only). Both keyed on the framework's lockfile so they invalidate on version bumps. - **No retry plumbing in the template.** Test System IO handles per-spec retries; the workflow only sees aggregated results. --- ## Pipeline 1: PR (`e2e-tests-ci.yml`) Runs E2E tests for every PR commit after the enterprise docker image is built. Fails if the commit is not associated with an open PR. **Trigger chain:** ``` PR commit ─► Enterprise CI builds docker image ─► Argo Events detects "Enterprise CI/docker-image" status ─► dispatches e2e-tests-ci.yml ``` For PRs from forks, `body.branches` may be empty so the workflow falls back to `master` for workflow files (trusted code), while `commit_sha` still points to the fork's commit. **Jobs:** 2 (cypress + playwright), each does smoke -> full **Commit statuses (4 total):** | Context | Description (pending) | Description (result) | |---------|----------------------|---------------------| | `e2e-test/cypress-smoke\|enterprise` | `tests running, image_tag:abc1234` | `100% passed (1313), 440 specs, image_tag:abc1234` | | `e2e-test/cypress-full\|enterprise` | `tests running, image_tag:abc1234` | `100% passed (1313), 440 specs, image_tag:abc1234` | | `e2e-test/playwright-smoke\|enterprise` | `tests running, image_tag:abc1234` | `100% passed (200), 50 specs, image_tag:abc1234` | | `e2e-test/playwright-full\|enterprise` | `tests running, image_tag:abc1234` | `99.5% passed (199/200), 1 failed, 50 specs, image_tag:abc1234` | **Manual trigger (CLI):** ```bash gh workflow run e2e-tests-ci.yml \ --repo mattermost/mattermost \ --field pr_number="35171" ``` **Manual trigger (GitHub UI):** 1. Go to **Actions** > **E2E Tests (smoke-then-full)** 2. Click **Run workflow** 3. Fill in `pr_number` (e.g., `35171`) 4. Click **Run workflow** ### On-demand testing For on-demand E2E testing, the existing triggers still work: - **Comment triggers**: `/e2e-test`, `/e2e-test fips`, or with `MM_ENV` parameters - **Label trigger**: `E2E/Run` These are separate from the automated workflow and can be used for custom test configurations or re-runs. --- ## Pipeline 2: Merge (`e2e-tests-on-merge.yml`) Runs E2E tests after every push/merge to `master` or `release-*` branches. **Trigger chain:** ``` Push to master/release-* ─► Argo Events (mattermost-platform-package sensor) ─► delivery-platform/.github/workflows/mattermost-platform-delivery.yaml ─► builds docker images (enterprise + fips) ─► trigger-e2e-tests job dispatches e2e-tests-on-merge.yml ``` **Jobs:** 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only **Commit statuses (4 total):** | Context | Description example | |---------|-------------------| | `e2e-test/cypress-full\|enterprise` | `100% passed (1313), 440 specs, image_tag:abc1234_def5678` | | `e2e-test/cypress-full\|fips` | `100% passed (1313), 440 specs, image_tag:abc1234_def5678` | | `e2e-test/playwright-full\|enterprise` | `100% passed (200), 50 specs, image_tag:abc1234_def5678` | | `e2e-test/playwright-full\|fips` | `100% passed (200), 50 specs, image_tag:abc1234_def5678` | **Manual trigger (CLI):** ```bash # For master gh workflow run e2e-tests-on-merge.yml \ --repo mattermost/mattermost \ --field branch="master" \ --field commit_sha="" \ --field server_image_tag="" # For release branch gh workflow run e2e-tests-on-merge.yml \ --repo mattermost/mattermost \ --field branch="release-11.4" \ --field commit_sha="" \ --field server_image_tag="" ``` **Manual trigger (GitHub UI):** 1. Go to **Actions** > **E2E Tests (master/release - merge)** 2. Click **Run workflow** 3. Fill in: - `branch`: `master` or `release-11.4` - `commit_sha`: full 40-char SHA - `server_image_tag`: e.g., `abc1234_def5678` 4. Click **Run workflow** --- ## Pipeline 3: Release Cut (`e2e-tests-on-release.yml`) Runs E2E tests after a release cut against the published release images. **Trigger chain:** ``` Manual release cut ─► delivery-platform/.github/workflows/release-mattermost-platform.yml ─► builds and publishes release docker images ─► trigger-e2e-tests job dispatches e2e-tests-on-release.yml ``` **Jobs:** 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only. Team edition planned for future. **Commit statuses (4 total, 6 when team is enabled):** Descriptions include alias tags showing which rolling docker tags point to the same image. RC example (11.4.0-rc3): | Context | Description example | |---------|-------------------| | `e2e-test/cypress-full\|enterprise` | `100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11)` | | `e2e-test/cypress-full\|fips` | `100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11)` | | `e2e-test/cypress-full\|team` (future) | `100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11)` | Stable example (11.4.0) — includes `MAJOR.MINOR` alias: | Context | Description example | |---------|-------------------| | `e2e-test/cypress-full\|enterprise` | `100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4)` | | `e2e-test/cypress-full\|fips` | `100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4)` | | `e2e-test/cypress-full\|team` (future) | `100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4)` | **Manual trigger (CLI):** ```bash gh workflow run e2e-tests-on-release.yml \ --repo mattermost/mattermost \ --field branch="release-11.4" \ --field commit_sha="" \ --field server_image_tag="11.4.0" \ --field server_image_aliases="release-11.4, release-11, 11.4" ``` **Manual trigger (GitHub UI):** 1. Go to **Actions** > **E2E Tests (release cut)** 2. Click **Run workflow** 3. Fill in: - `branch`: `release-11.4` - `commit_sha`: full 40-char SHA - `server_image_tag`: e.g., `11.4.0` or `11.4.0-rc3` - `server_image_aliases`: e.g., `release-11.4, release-11, 11.4` (optional) 4. Click **Run workflow** --- ## Commit Status Format **Context name:** `e2e-test/|` Where `` is `cypress-smoke`, `cypress-full`, `playwright-smoke`, or `playwright-full`. **Description format:** - All passed: `100% passed (), specs, image_tag:[ ()]` - With failures: `% passed (/), failed, specs, image_tag:[ ()]` - Pending: `tests running, image_tag:[ ()]` - Pass rate: `100%` if all pass, otherwise one decimal (e.g., `99.5%`) - Aliases only present for release cuts ### Failure behavior 1. **Smoke test fails**: Full tests are skipped, only smoke commit status shows failure 2. **Full test fails**: Full commit status shows failure with pass rate 3. **Both pass**: Both smoke and full commit statuses show success 4. **No PR found** (PR pipeline only): Workflow fails immediately --- ## Smoke-then-Full Pattern Each wrapper (Cypress/Playwright) follows this flow: ``` generate-build-variables (branch, build_id, server_image) ─► smoke tests (1 worker, minimal docker services) ─► if smoke passes ─► full tests (20 workers cypress / 1 worker playwright, all docker services) ─► report (aggregate results, update commit status) ``` ### Test filtering | Framework | Smoke | Full | |-----------|-------|------| | **Cypress** | `--stage=@prod --group=@smoke` | `--stage="@prod" --excludeGroup="@te_only,@cloud_only,@high_availability" --sortFirst=... --sortLast=...` | | **Playwright** | `--grep @smoke` | `--grep-invert "@smoke\|@visual"` | ### Worker configuration | Framework | Smoke Workers | Full Workers | |-----------|---------------|--------------| | **Cypress** | 1 | 20 | | **Playwright** | 1 | 1 (uses internal parallelism via `PW_WORKERS`) | ### Docker services | Test Phase | Docker Services | |------------|-----------------| | Smoke | `postgres inbucket` | | Full | `postgres inbucket minio openldap elasticsearch keycloak` | --- ## Tagging Smoke Tests ### Cypress Add `@smoke` to the Group comment at the top of spec files: ```javascript // Stage: @prod // Group: @channels @messaging @smoke ``` ### Playwright Add `@smoke` to the test tag option: ```typescript test('critical login flow', {tag: ['@smoke', '@login']}, async ({pw}) => { // ... }); ``` --- ## Shared Wrapper Inputs The wrappers (`e2e-tests-cypress.yml`, `e2e-tests-playwright.yml`) accept these inputs: | Input | Default | Description | |-------|---------|-------------| | `server_edition` | `enterprise` | Edition: `enterprise`, `fips`, or `team` | | `server_image_repo` | `mattermostdevelopment` | Docker namespace: `mattermostdevelopment` or `mattermost` | | `server_image_tag` | derived from `commit_sha` | Docker image tag | | `server_image_aliases` | _(empty)_ | Alias tags shown in commit status description | | `ref_branch` | _(empty)_ | Source branch name for webhook messages (e.g., `master` or `release-11.4`) | The automation dashboard branch name is derived from context: - PR: `pr-` (e.g., `pr-35205`) - Master merge: `master` - Release merge: `release-` (e.g., `release-11.7`) - Fallback: `commit-` (e.g., `commit-abc1234`) Master and release runs use the real branch verbatim so the `/reports/{repo}/{branch}` view aggregates every build on that branch into one history. PR runs use a `pr-` token because there's no real branch to use; the commit-only fallback uses a `commit-` token (defensive — no current caller hits this path). The test type suffix (`-smoke` or `-full`) is appended by the template. The server image is derived as: ``` {server_image_repo}/{edition_image_name}:{server_image_tag} ``` Where `edition_image_name` maps to: - `enterprise` -> `mattermost-enterprise-edition` - `fips` -> `mattermost-enterprise-fips-edition` - `team` -> `mattermost-team-edition` --- ## Webhook Message Format After full tests complete, a webhook notification is sent to the configured `REPORT_WEBHOOK_URL`. The results line uses the same `commit_status_message` as the GitHub commit status. The source line varies by pipeline using `report_type` and `ref_branch`. **Report types:** `PR`, `MASTER`, `RELEASE`, `RELEASE_CUT` ### PR ``` :open-pull-request: mattermost-pr-35205 :docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234 100% passed (1313), 440 specs | full report ``` ### Merge to master ``` :git_merge: abc1234 on master :docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678 100% passed (1313), 440 specs | full report ``` ### Merge to release branch ``` :git_merge: abc1234 on release-11.4 :docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678 100% passed (1313), 440 specs | full report ``` ### Release cut ``` :github_round: abc1234 on release-11.4 :docker: mattermost/mattermost-enterprise-edition:11.4.0-rc3 100% passed (1313), 440 specs | full report ``` The commit short SHA links to the commit on GitHub. The PR number links to the pull request. --- ## Related Files - `e2e-tests/cypress/` - Cypress test suite - `e2e-tests/playwright/` - Playwright test suite - `e2e-tests/.ci/` - CI configuration and environment files - `e2e-tests/Makefile` - Makefile with targets for running tests, generating cycles, and reporting