ci: shard server Postgres tests into 4 parallel runners (#35739)

* ci: add test sharding plumbing to server CI

Add infrastructure for upcoming test sharding without changing behavior:

- Add shard-index and shard-total inputs to server-test-template.yml
  (defaults preserve existing single-runner behavior)
- Add timing cache restore step (activates only when shard-total > 1)
- Add merge-postgres-test-results job to server-ci.yml that:
  - Merges JUnit XML reports from shard artifacts
  - Saves timing data cache for future shard balancing
  - Handles both single-artifact and multi-shard scenarios
- Add .gitignore entries for timing cache and shard work files

* ci: shard server Postgres tests into 4 parallel runners

Extract sharding logic into standalone, tested scripts and enable
4-shard parallel test execution for server Postgres CI:

Scripts:
- server/scripts/shard-split.js: Node.js bin-packing solver that
  assigns test packages to shards using timing data from previous runs.
  Two-tier strategy: light packages (<2min) whole, heavy packages
  (api4, app) split at individual test level.
- server/scripts/run-shard-tests.sh: Multi-run wrapper that calls
  gotestsum directly for each package group with -run regex filters.
- server/scripts/shard-split.test.js: 8 test cases covering round-robin
  fallback, timing-based balancing, heavy package splitting, JUnit XML
  fallback, and enterprise package separation.

Workflow changes:
- server-test-template.yml: Add shard splitting step that discovers test
  packages and runs the solver. Modified Run Tests step to use wrapper
  script when sharding is active.
- server-ci.yml: Add 4-shard matrix to test-postgres-normal. Update
  merge job artifact patterns for shard-specific names.

Performance: 7.2 min with timing cache vs 62.5 min baseline = 88%
wall-time improvement. First run without cache uses JUnit XML fallback
or round-robin, then populates the cache for subsequent runs.

* fix: raise heavy package threshold to 5 min to preserve test isolation

sqlstore integrity tests scan the entire database and fail when other
packages' test data is present. At 182s, sqlstore was just over the
120s threshold and getting split at test level. Raising to 300s keeps
only api4 (~38 min) and app (~15 min) as heavy — where the real
sharding gains are — while sqlstore, elasticsearch, etc. stay whole
and maintain their test isolation guarantees.

* ci: only save test timing cache on default branch

PR branches always restore from master's timing cache via restore-keys
prefix matching. Timing data is stable day-to-day so this eliminates
cache misses on first PR runs and reduces cache storage.

* ci: skip FIPS tests on PRs (enterprise CI handles compile check)

Per review feedback: the enterprise CI already runs a FIPS compile
check on every PR. Running the full FIPS test suite on PRs is redundant
since it uses the identical test suite as non-FIPS — the only
FIPS-specific failure mode is a build failure from non-approved crypto
imports, which the enterprise compile check catches.

Full FIPS tests continue to run on every push to master.

* fix: address review feedback on run-shard-tests.sh

- Remove set -e so all test runs execute even if earlier ones fail;
  track failures and exit with error at the end (wiggin77)
- Remove unused top-level COVERAGE_FLAG variable (wiggin77)
- Fix RUN_IDX increment position so report, json, and coverage files
  share the same index (wiggin77)
- Update workflow comment: heavy threshold is 5 min, not 2 min (wiggin77)

* style: use node: prefix for built-in fs module in shard-split.js

* fix: avoid interpolating file paths into generated shell script

Read shard package lists from files at runtime instead of interpolating
them into the generated script via printf. This prevents theoretical
shell metacharacter injection from directory names, as flagged by
DryRun Security.

* fix(ci): rename merged artifact to match server-ci-report glob

The merged artifact was named postgres-server-test-logs-merged which
does not match the *-test-logs pattern in server-ci-report.yml,
causing Postgres test results to be missing from PR/commit reports.

Also pins junit-report-merger to exact version 7.0.0 for supply chain
safety.

* fix(ci): pass RACE_MODE env into Docker container

RACE_MODE was set on the host runner but never included in the docker
run --env list. The light-package path worked because the heredoc
expanded on the host, but run-shard-tests.sh reads RACE_MODE at
runtime inside the container where it was unset. This caused heavy
packages (api4, app) to silently lose -race detection.

* fix(ci): discover new tests in heavy packages not in timing cache

Tests not present in the timing cache (newly added or renamed) would
not appear in any shard -run regex, causing them to silently skip.
After building items from the cache, run go test -list to discover
current test names and assign any cache-missing tests to shards via
the normal bin-packing algorithm with a small default duration.

* fix(ci): add missing line continuation backslash in docker run

The previous --env FIPS_ENABLED line was missing a trailing backslash
after adding --env RACE_MODE, causing docker run to see a truncated
command and fail with "requires at least 1 argument".

* fix(ci): add setup-go step for shard test discovery

go test -list in shard-split.js runs on the host runner via execSync,
but Go is only available inside the Docker container. Without this
step, every invocation fails silently and new-test discovery is a
no-op. Adding actions/setup-go before the shard split step ensures
the Go toolchain is available on the host.

---------

Co-authored-by: Hanzo Dev <[email protected]>

.github/workflows/server-ci.yml

@@ -199,21 +199,103 @@ jobs:
       logsartifact: postgres-binary-server-test-logs
       go-version: ${{ needs.go.outputs.version }}
       fips-enabled: false
+  # -- Sharded into 4 parallel runners for ~88% wall-time improvement --
   test-postgres-normal:
-    name: Postgres
+    name: Postgres (shard ${{ matrix.shard }})
     needs: go
+    strategy:
+      fail-fast: false # Let all shards complete so we get full test results
+      matrix:
+        shard: [0, 1, 2, 3]
     uses: ./.github/workflows/server-test-template.yml
     secrets: inherit
     with:
-      name: Postgres
+      name: "Postgres (shard ${{ matrix.shard }})"
       datasource: postgres://mmuser:mostest@postgres:5432/mattermost_test?sslmode=disable&connect_timeout=10
       drivername: postgres
-      logsartifact: postgres-server-test-logs
+      # Each shard gets a unique artifact name so they don't collide
+      logsartifact: "postgres-server-test-logs-shard-${{ matrix.shard }}"
       go-version: ${{ needs.go.outputs.version }}
       fips-enabled: false
+      shard-index: ${{ matrix.shard }}
+      shard-total: 4
+  # -- Merge test results (handles both single-run and future sharded runs) --
+  merge-postgres-test-results:
+    name: Merge Postgres Test Results
+    needs: test-postgres-normal
+    if: always()
+    runs-on: ubuntu-22.04
+    steps:
+      - name: Download all shard artifacts
+        uses: actions/download-artifact@70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3 # v8.0.0
+        with:
+          pattern: postgres-server-test-logs-shard-*
+          path: shards
+
+      - name: Setup Node for junit-report-merger
+        uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
+        with:
+          node-version: 20
+
+      - name: Merge JUnit reports
+        run: |
+          npx [email protected] merged-report.xml "shards/*/report.xml"
+
+      # Prepare files with expected names for the report workflow
+      - name: Prepare merged artifact
+        run: |
+          mkdir -p merged
+          cp merged-report.xml merged/report.xml
+          # Find the first shard artifact that has test-name and pr-number
+          for dir in shards/postgres-server-test-logs-shard-*/; do
+            if [[ -f "${dir}test-name" ]]; then
+              cp "${dir}test-name" merged/test-name
+              cp "${dir}pr-number" merged/pr-number
+              break
+            fi
+          done
+
+      # Upload merged artifact with the name the report workflow expects
+      - name: Upload merged test logs
+        uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
+        with:
+          name: postgres-server-test-logs
+          path: merged/
+
+      # Save merged timing data for future shard balancing.
+      # Includes both JUnit XML (package-level) and gotestsum.json (per-test).
+      - name: Prepare timing cache
+        id: timing-prep
+        run: |
+          mkdir -p server
+          # Only save timing cache if we have real data (merged report > 1KB).
+          # Failed runs produce tiny/empty files that would overwrite good data.
+          if [[ -f merged-report.xml && $(stat -c%s merged-report.xml) -gt 1024 ]]; then
+            cp merged-report.xml server/prev-report.xml
+            # Merge gotestsum.json (JSONL) from all shards for per-test timing
+            cat shards/*/gotestsum.json > server/prev-gotestsum.json 2>/dev/null || true
+            echo "has_timing=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "Skipping timing cache — merged report too small or missing"
+            echo "has_timing=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Save test timing cache
+        # Only save timing on the default branch. PR branches restore from
+        # master via restore-keys prefix match. Timing data is stable day-to-day
+        # so this eliminates cache misses on first PR runs.
+        if: steps.timing-prep.outputs.has_timing == 'true' && github.ref_name == github.event.repository.default_branch
+        uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+        with:
+          path: |
+            server/prev-report.xml
+            server/prev-gotestsum.json
+          key: server-test-timing-master-${{ github.run_id }}
+
   test-postgres-normal-fips:
-    # Skip FIPS testing for forks, which won't have docker login credentials.
-    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
+    # FIPS compile check already runs in enterprise CI for PRs.
+    # Only run full FIPS tests on pushes to the default branch.
+    if: github.event_name == 'push'
     name: Postgres (FIPS)
     needs: go
     uses: ./.github/workflows/server-test-template.yml

.github/workflows/server-test-template.yml

@@ -29,6 +29,15 @@ on:
         required: false
         default: false
         type: boolean
+      # -- Test sharding inputs (leave defaults for non-sharded callers) --
+      shard-index:
+        required: false
+        type: number
+        default: -1 # -1 = no sharding; run all tests
+      shard-total:
+        required: false
+        type: number
+        default: 1
 
 permissions:
   id-token: write
@@ -52,6 +61,21 @@ jobs:
 
       - name: Checkout mattermost project
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Restore test timing data
+        if: inputs.shard-total > 1
+        id: timing-cache
+        uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+        with:
+          path: |
+            server/prev-report.xml
+            server/prev-gotestsum.json
+          # Always restore from master — timing is only saved on the default
+          # branch and is stable enough for shard balancing.
+          key: server-test-timing-master
+          restore-keys: |
+            server-test-timing-
+
       - name: Setup BUILD_IMAGE
         id: build
         run: |
@@ -78,6 +102,60 @@ jobs:
           docker compose --ansi never exec -T minio sh -c 'mkdir -p /data/mattermost-test';
           docker compose --ansi never ps
 
+      # ── Test-level sharding ────────────────────────────────────────────
+      # When shard-total > 1, we split tests across N parallel runners.
+      #
+      # Two-tier splitting strategy:
+      #   - "Light" packages (< 5 min): assigned whole to a shard
+      #   - "Heavy" packages (≥ 5 min, e.g. api4, app): individual tests
+      #     are distributed across shards using -run regex filters
+      #
+      # See server/scripts/shard-split.js for the full algorithm.
+      # ─────────────────────────────────────────────────────────────────────
+
+      - name: Setup Go for test discovery
+        if: inputs.shard-total > 1
+        uses: actions/setup-go@4b73464bb391d4059bd26b0524d20df3927bd417 # v6.3.0
+        with:
+          go-version: ${{ inputs.go-version }}
+
+      - name: Split tests across shards
+        if: inputs.shard-total > 1
+        id: test_split
+        working-directory: server
+        env:
+          SHARD_INDEX: ${{ inputs.shard-index }}
+          SHARD_TOTAL: ${{ inputs.shard-total }}
+        run: |
+          set -euo pipefail
+
+          # ── List all test packages ──
+          echo "::group::Listing test packages"
+          TE_PKGS=$(find ./public/ ./ -name '*_test.go' -not -path './enterprise/*' -not -path './cmd/mmctl/*' 2>/dev/null \
+            | sed 's|/[^/]*$||' | sort -u \
+            | sed 's|^\./|github.com/mattermost/mattermost/server/v8/|' \
+            | sed 's|github.com/mattermost/mattermost/server/v8/public/|github.com/mattermost/mattermost/server/public/|')
+          EE_PKGS=$(find ./enterprise/ -name '*_test.go' 2>/dev/null \
+            | sed 's|/[^/]*$||' | sort -u \
+            | sed 's|^\./|github.com/mattermost/mattermost/server/v8/|')
+          ALL_PKGS=$(printf '%s\n%s' "$TE_PKGS" "$EE_PKGS" | grep -v '^$' | sort -u)
+          TOTAL_PKGS=$(echo "$ALL_PKGS" | wc -l)
+          echo "Found $TOTAL_PKGS test packages"
+          echo "::endgroup::"
+
+          if [[ "$TOTAL_PKGS" -eq 0 ]]; then
+            echo "WARNING: No test packages found"
+            echo "has_packages=false" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          echo "$ALL_PKGS" > all-packages.txt
+
+          # ── Run shard solver ──
+          node scripts/shard-split.js
+
+          echo "has_packages=true" >> "$GITHUB_OUTPUT"
+
       - name: Run Tests
         env:
           BUILD_IMAGE: ${{ steps.build.outputs.BUILD_IMAGE }}
@@ -85,6 +163,38 @@ jobs:
           if [[ ${{ github.ref_name }} == 'master' && ${{ inputs.fullyparallel }} != true ]]; then
             export RACE_MODE="-race"
           fi
+
+          MAKE_ARGS="test-server${RACE_MODE} BUILD_NUMBER=${GITHUB_HEAD_REF}-${GITHUB_RUN_ID}"
+          DOCKER_CMD="make ${MAKE_ARGS}"
+
+          # When sharding is active, use the multi-run wrapper script
+          if [[ "${{ inputs.shard-total }}" -gt 1 && -f server/shard-te-packages.txt ]]; then
+            SHARD_TE=$(cat server/shard-te-packages.txt)
+            SHARD_EE=$(cat server/shard-ee-packages.txt)
+            HEAVY_RUNS=""
+            if [[ -f server/shard-heavy-runs.txt && -s server/shard-heavy-runs.txt ]]; then
+              HEAVY_RUNS=$(cat server/shard-heavy-runs.txt)
+            fi
+
+            if [[ -z "$HEAVY_RUNS" ]]; then
+              # No heavy packages — single run via Makefile with package filter.
+              # Read packages from files at runtime to avoid interpolating
+              # file-system-derived paths into a generated shell script.
+              cat > server/run-shard-tests.sh <<SHARD_EOF
+          #!/bin/bash
+          exec make ${MAKE_ARGS} \\
+            TE_PACKAGES="\$(cat shard-te-packages.txt)" \\
+            EE_PACKAGES="\$(cat shard-ee-packages.txt)"
+          SHARD_EOF
+            else
+              # Use the multi-run wrapper script
+              cp server/scripts/run-shard-tests.sh server/run-shard-tests.sh
+            fi
+
+            chmod +x server/run-shard-tests.sh
+            DOCKER_CMD="/mattermost/server/run-shard-tests.sh"
+          fi
+
           docker run --net ghactions_mm-test \
             --ulimit nofile=8096:8096 \
             --env-file=server/build/dotenv/test.env \
@@ -94,10 +204,11 @@ jobs:
             --env ENABLE_FULLY_PARALLEL_TESTS="${{ inputs.fullyparallel }}" \
             --env ENABLE_COVERAGE="${{ inputs.enablecoverage }}" \
             --env FIPS_ENABLED="${{ inputs.fips-enabled }}" \
+            --env RACE_MODE \
             -v $PWD:/mattermost \
             -w /mattermost/server \
             $BUILD_IMAGE \
-            make test-server$RACE_MODE BUILD_NUMBER=$GITHUB_HEAD_REF-$GITHUB_RUN_ID
+            $DOCKER_CMD
       - name: Upload coverage to Codecov
         if: ${{ inputs.enablecoverage }}
         uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
@@ -122,3 +233,4 @@ jobs:
             server/cover.out
             server/test-name
             server/pr-number
+

.gitignore

@@ -166,3 +166,6 @@ docker-compose.override.yaml
 AGENTS.md
 .cursorrules
 .cursor/
+server/prev-report.xml
+server/prev-gotestsum.json
+server/shard-*.txt

server/scripts/run-shard-tests.sh

@@ -0,0 +1,116 @@
+#!/bin/bash
+set -uo pipefail
+
+# run-shard-tests.sh — Multi-run test wrapper for sharded CI
+#
+# When a shard has both "light" packages (run whole) and "heavy" package
+# splits (run with -run regex), we need multiple gotestsum invocations.
+# The Makefile's test-server target only supports a single invocation,
+# so this script calls gotestsum directly.
+#
+# Each invocation produces its own JUnit XML and JSON log files, which
+# are merged at the end into the standard report.xml and gotestsum.json
+# that the CI pipeline expects.
+#
+# Input files (in working directory, written by shard-split.js):
+#   shard-te-packages.txt   — space-separated TE packages
+#   shard-ee-packages.txt   — space-separated EE packages
+#   shard-heavy-runs.txt    — one line per heavy run: "pkg REGEX"
+#
+# Environment variables (set by CI):
+#   RACE_MODE          — "-race" on master, empty on PRs
+#   ENABLE_COVERAGE    — "true" to enable coverage profiling
+
+GOBIN="$(pwd)/bin"
+
+# Set up build prerequisites (go.work, gotestsum, go versions)
+# These are normally done by make test-server-pre.
+make setup-go-work gotestsum golang-versions
+
+GOFLAGS_BASE="-buildvcs=false -timeout=90m"
+RACE_FLAG="${RACE_MODE:-}"
+
+RUN_IDX=0
+FAILURES=0
+
+# run_gotestsum PACKAGES [RUN_REGEX]
+#   $1 = space-separated package list
+#   $2 = optional -run regex (passed directly to go test)
+run_gotestsum() {
+  local junitfile="report-${RUN_IDX}.xml"
+  local jsonfile="gotestsum-${RUN_IDX}.json"
+  local run_flag=""
+  if [[ -n "${2:-}" ]]; then run_flag="-run $2"; fi
+
+  local coverage_flag=""
+  if [[ "${ENABLE_COVERAGE:-false}" == "true" ]]; then
+    coverage_flag="-coverprofile=cover-${RUN_IDX}.out -covermode=atomic"
+  fi
+
+  RUN_IDX=$((RUN_IDX + 1))
+
+  GOTESTSUM_JUNITFILE="$junitfile" GOTESTSUM_JSONFILE="$jsonfile" \
+    "$GOBIN/gotestsum" --rerun-fails=3 --packages="$1" \
+    -- $GOFLAGS_BASE $RACE_FLAG $coverage_flag $run_flag \
+    || FAILURES=$((FAILURES + 1))
+}
+
+# ── Read shard assignments ──
+SHARD_TE=""
+SHARD_EE=""
+HEAVY_RUNS=""
+
+if [[ -f shard-te-packages.txt ]]; then
+  SHARD_TE=$(cat shard-te-packages.txt)
+fi
+if [[ -f shard-ee-packages.txt ]]; then
+  SHARD_EE=$(cat shard-ee-packages.txt)
+fi
+if [[ -f shard-heavy-runs.txt && -s shard-heavy-runs.txt ]]; then
+  HEAVY_RUNS=$(cat shard-heavy-runs.txt)
+fi
+
+# ── Run light packages (single invocation, no -run filter) ──
+ALL_LIGHT="${SHARD_TE} ${SHARD_EE}"
+ALL_LIGHT="${ALL_LIGHT## }"
+ALL_LIGHT="${ALL_LIGHT%% }"
+if [[ -n "$ALL_LIGHT" ]]; then
+  LIGHT_COUNT=$(echo "$ALL_LIGHT" | wc -w)
+  echo "Running $LIGHT_COUNT light packages..."
+  run_gotestsum "$ALL_LIGHT"
+fi
+
+# ── Run heavy package splits (one invocation per package subset) ──
+if [[ -n "$HEAVY_RUNS" ]]; then
+  while IFS= read -r line; do
+    [[ -z "$line" ]] && continue
+    PKG="${line%% *}"
+    REGEX="${line#* }"
+    SHORT_PKG="${PKG##*/}"
+    TEST_COUNT=$(echo "$REGEX" | tr '|' '\n' | wc -l)
+    echo "Running $TEST_COUNT tests from $SHORT_PKG..."
+    run_gotestsum "$PKG" "$REGEX"
+  done <<< "$HEAVY_RUNS"
+fi
+
+# ── Merge results from all runs ──
+echo "Merging results from $RUN_IDX gotestsum runs..."
+
+if ls report-*.xml 1>/dev/null 2>&1; then
+  # Simple XML concatenation — the merge job uses junit-report-merger for proper merging
+  head -1 report-0.xml > report.xml
+  echo "<testsuites>" >> report.xml
+  for f in report-*.xml; do
+    grep -v "<?xml" "$f" | grep -v "^<testsuites" | grep -v "^</testsuites" >> report.xml || true
+  done
+  echo "</testsuites>" >> report.xml
+fi
+
+cat gotestsum-*.json > gotestsum.json 2>/dev/null || true
+
+if [[ $FAILURES -gt 0 ]]; then
+  echo "Shard complete: $RUN_IDX gotestsum runs, $FAILURES failed"
+  exit 1
+fi
+
+echo "Shard complete: $RUN_IDX gotestsum runs, all passed"

server/scripts/shard-split.js

@@ -0,0 +1,242 @@
+#!/usr/bin/env node
+/**
+ * shard-split.js — Test shard assignment solver
+ *
+ * Splits Go test packages across N parallel CI runners using timing data
+ * from previous runs. Uses a two-tier strategy:
+ *
+ *   1. "Light" packages (< HEAVY_MS total runtime): assigned whole to a shard
+ *   2. "Heavy" packages (>= HEAVY_MS): individual tests distributed across
+ *      shards using -run regex filters
+ *
+ * Timing data sources (in priority order):
+ *   - gotestsum.json (JSONL): per-test elapsed times from previous run
+ *   - prev-report.xml (JUnit XML): package-level timing (fallback)
+ *   - Round-robin: when no timing data exists at all
+ *
+ * Assignment algorithm: greedy bin-packing (sort by duration desc, assign
+ * each item to the shard with lowest current load). Simple and effective
+ * for our distribution where 2 packages dominate 84% of runtime.
+ *
+ * Environment variables:
+ *   SHARD_INDEX  — this runner's index (0-based)
+ *   SHARD_TOTAL  — total number of shards
+ *
+ * Input files (in working directory):
+ *   all-packages.txt      — newline-separated list of all test packages
+ *   prev-gotestsum.json   — (optional) JSONL timing data from previous run
+ *   prev-report.xml       — (optional) JUnit XML from previous run
+ *
+ * Output files (in working directory):
+ *   shard-te-packages.txt   — space-separated TE packages for this shard
+ *   shard-ee-packages.txt   — space-separated EE packages for this shard
+ *   shard-heavy-runs.txt    — heavy package runs, one per line: "pkg REGEX"
+ */
+
+const fs = require("node:fs");
+const { execSync } = require("node:child_process");
+
+const SHARD_INDEX = parseInt(process.env.SHARD_INDEX);
+const SHARD_TOTAL = parseInt(process.env.SHARD_TOTAL);
+const HEAVY_MS = 300000; // 5 min: packages above this get test-level splitting
+// Only api4 (~38 min) and app (~15 min) exceed this threshold.
+// Packages like sqlstore (~3 min) stay whole to preserve test isolation —
+// their integrity tests scan the entire database and break if split across
+// shards where other tests leave data behind.
+
+if (isNaN(SHARD_INDEX) || isNaN(SHARD_TOTAL) || SHARD_TOTAL < 1) {
+  console.error("ERROR: SHARD_INDEX and SHARD_TOTAL must be set");
+  process.exit(1);
+}
+
+const allPkgs = fs.readFileSync("all-packages.txt", "utf8").trim().split("\n").filter(Boolean);
+if (allPkgs.length === 0) {
+  console.error("WARNING: No test packages found in all-packages.txt");
+  process.exit(0);
+}
+
+const pkgTimes = {};
+const testTimes = {}; // "pkg::TestName" -> ms
+
+// ── Parse gotestsum.json (JSONL) for per-test timing ──
+// Each line is a JSON event; we want "pass" events with Elapsed times.
+if (fs.existsSync("prev-gotestsum.json")) {
+  console.log("::group::Parsing gotestsum.json timing data");
+  const lines = fs.readFileSync("prev-gotestsum.json", "utf8").split("\n");
+  for (const line of lines) {
+    if (!line.includes('"pass"')) continue;
+    try {
+      const d = JSON.parse(line);
+      if (!d.Test || !d.Package) continue;
+      const elapsed = Math.round((d.Elapsed || 0) * 1000);
+      // Aggregate package time from test pass events
+      pkgTimes[d.Package] = (pkgTimes[d.Package] || 0) + elapsed;
+      // Top-level test name (use max elapsed for parent vs subtests)
+      const top = d.Test.split("/")[0];
+      const key = d.Package + "::" + top;
+      testTimes[key] = Math.max(testTimes[key] || 0, elapsed);
+    } catch (e) {
+      // Skip malformed lines
+    }
+  }
+  console.log(
+    `gotestsum.json: ${Object.keys(pkgTimes).length} packages, ${Object.keys(testTimes).length} tests`
+  );
+  console.log("::endgroup::");
+}
+
+// ── Fallback: parse JUnit XML for package-level timing ──
+if (Object.keys(pkgTimes).length === 0 && fs.existsSync("prev-report.xml")) {
+  console.log("::group::Parsing JUnit XML timing data (fallback)");
+  const xml = fs.readFileSync("prev-report.xml", "utf8");
+  for (const m of xml.matchAll(/<testsuite[^>]*>/g)) {
+    const name = m[0].match(/name="([^"]+)"/)?.[1];
+    const time = m[0].match(/\btime="([^"]+)"/)?.[1];
+    if (name && time) {
+      pkgTimes[name] = (pkgTimes[name] || 0) + Math.round(parseFloat(time) * 1000);
+    }
+  }
+  console.log(`JUnit XML: ${Object.keys(pkgTimes).length} packages (no per-test data)`);
+  console.log("::endgroup::");
+}
+
+const hasTimingData = Object.keys(pkgTimes).length > 0;
+const hasTestTiming = Object.keys(testTimes).length > 0;
+
+// ── Identify heavy packages ──
+// Only split at test level if we have per-test timing data
+const heavyPkgs = new Set();
+if (hasTestTiming) {
+  for (const [pkg, ms] of Object.entries(pkgTimes)) {
+    if (ms > HEAVY_MS) heavyPkgs.add(pkg);
+  }
+}
+if (heavyPkgs.size > 0) {
+  console.log("Heavy packages (test-level splitting):");
+  for (const p of heavyPkgs) {
+    console.log(`  ${(pkgTimes[p] / 1000).toFixed(0)}s  ${p.split("/").pop()}`);
+  }
+}
+
+// ── Build work items ──
+// Each item is either a whole package ("P") or a single test from a heavy package ("T")
+const items = [];
+for (const pkg of allPkgs) {
+  if (heavyPkgs.has(pkg)) {
+    // Split into individual test items
+    const tests = Object.entries(testTimes)
+      .filter(([k]) => k.startsWith(pkg + "::"))
+      .map(([k, ms]) => ({ ms, type: "T", pkg, test: k.split("::")[1] }));
+    if (tests.length > 0) {
+      items.push(...tests);
+    } else {
+      // Shouldn't happen, but fall back to whole package
+      items.push({ ms: pkgTimes[pkg] || 1, type: "P", pkg });
+    }
+  } else {
+    items.push({ ms: pkgTimes[pkg] || 1, type: "P", pkg });
+  }
+}
+// ── Discover new/renamed tests in heavy packages ──
+// Tests not in the timing cache won't appear in any shard's -run regex,
+// silently skipping them. Discover current test names at runtime and
+// assign any cache-missing tests to the least-loaded shard.
+if (heavyPkgs.size > 0) {
+  console.log("::group::Discovering new tests in heavy packages");
+  for (const pkg of heavyPkgs) {
+    const cachedTests = new Set(
+      Object.keys(testTimes)
+        .filter((k) => k.startsWith(pkg + "::"))
+        .map((k) => k.split("::")[1])
+    );
+    try {
+      const out = execSync(`go test -list '.*' ${pkg} 2>/dev/null`, {
+        encoding: "utf8",
+        timeout: 60000,
+      });
+      const currentTests = out
+        .split("\n")
+        .map((l) => l.trim())
+        .filter((l) => /^Test[A-Z]/.test(l));
+      let newCount = 0;
+      for (const t of currentTests) {
+        if (!cachedTests.has(t)) {
+          // Assign a small default duration so it gets picked up
+          items.push({ ms: 1000, type: "T", pkg, test: t });
+          newCount++;
+        }
+      }
+      if (newCount > 0) {
+        console.log(`  ${pkg.split("/").pop()}: ${newCount} new test(s) not in cache`);
+      }
+    } catch (e) {
+      console.log(`  ${pkg.split("/").pop()}: go test -list failed, skipping discovery`);
+    }
+  }
+  console.log("::endgroup::");
+}
+
+// Sort descending by duration for greedy bin-packing
+items.sort((a, b) => b.ms - a.ms);
+
+// ── Greedy bin-packing assignment ──
+const shards = Array.from({ length: SHARD_TOTAL }, () => ({
+  load: 0,
+  whole: [],
+  heavy: {},
+}));
+
+if (!hasTimingData) {
+  // Round-robin fallback when no timing data exists
+  console.log("No timing data — using round-robin");
+  allPkgs.forEach((pkg, i) => {
+    shards[i % SHARD_TOTAL].whole.push(pkg);
+  });
+} else {
+  for (const item of items) {
+    // Find shard with minimum current load
+    const min = shards.reduce((m, s, i) => (s.load < shards[m].load ? i : m), 0);
+    shards[min].load += item.ms;
+    if (item.type === "P") {
+      shards[min].whole.push(item.pkg);
+    } else {
+      if (!shards[min].heavy[item.pkg]) shards[min].heavy[item.pkg] = [];
+      shards[min].heavy[item.pkg].push(item.test);
+    }
+  }
+}
+
+// ── Report shard assignments ──
+console.log("::group::Shard assignment");
+for (let i = 0; i < SHARD_TOTAL; i++) {
+  const s = shards[i];
+  const hRuns = Object.keys(s.heavy).length;
+  const hTests = Object.values(s.heavy).reduce((n, a) => n + a.length, 0);
+  const marker = i === SHARD_INDEX ? " ← THIS SHARD" : "";
+  console.log(
+    `Shard ${i}: ${(s.load / 1000).toFixed(1)}s | ${s.whole.length} pkgs` +
+      (hRuns > 0 ? `, ${hRuns} heavy splits (${hTests} tests)` : "") +
+      marker
+  );
+}
+console.log("::endgroup::");
+
+// ── Write output for this shard ──
+const myShard = shards[SHARD_INDEX];
+const te = myShard.whole.filter((p) => !p.includes("/enterprise/")).join(" ");
+const ee = myShard.whole.filter((p) => p.includes("/enterprise/")).join(" ");
+
+fs.writeFileSync("shard-te-packages.txt", te);
+fs.writeFileSync("shard-ee-packages.txt", ee);
+
+// Heavy package runs: one line per run as "pkg REGEX"
+const heavyRuns = Object.entries(myShard.heavy).map(([pkg, tests]) => {
+  const regex = tests.map((t) => "^" + t + "$").join("|");
+  return pkg + " " + regex;
+});
+fs.writeFileSync("shard-heavy-runs.txt", heavyRuns.join("\n"));
+
+console.log(
+  `Light packages: ${myShard.whole.length} (${te.split(" ").filter(Boolean).length} TE, ${ee.split(" ").filter(Boolean).length} EE)`
+);
+console.log(`Heavy package runs: ${heavyRuns.length}`);

server/scripts/shard-split.test.js

@@ -0,0 +1,333 @@
+const { describe, it, beforeEach, afterEach } = require("node:test");
+const assert = require("node:assert/strict");
+const fs = require("node:fs");
+const path = require("node:path");
+const { execFileSync } = require("node:child_process");
+const os = require("node:os");
+
+const SCRIPT = path.join(__dirname, "shard-split.js");
+const TESTDATA = path.join(__dirname, "testdata");
+
+/**
+ * Helper: run shard-split.js in a temp directory with given inputs.
+ * Returns the output files and stdout.
+ */
+function runSolver({ packages, shardIndex, shardTotal, gotestsumJson, prevReportXml }) {
+  const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "shard-test-"));
+  try {
+    fs.writeFileSync(path.join(tmpDir, "all-packages.txt"), packages.join("\n"));
+
+    if (gotestsumJson) {
+      fs.writeFileSync(path.join(tmpDir, "prev-gotestsum.json"), gotestsumJson);
+    }
+    if (prevReportXml) {
+      fs.writeFileSync(path.join(tmpDir, "prev-report.xml"), prevReportXml);
+    }
+
+    const stdout = execFileSync("node", [SCRIPT], {
+      cwd: tmpDir,
+      env: {
+        ...process.env,
+        SHARD_INDEX: String(shardIndex),
+        SHARD_TOTAL: String(shardTotal),
+      },
+      encoding: "utf8",
+    });
+
+    const te = fs.readFileSync(path.join(tmpDir, "shard-te-packages.txt"), "utf8");
+    const ee = fs.readFileSync(path.join(tmpDir, "shard-ee-packages.txt"), "utf8");
+    const heavy = fs.readFileSync(path.join(tmpDir, "shard-heavy-runs.txt"), "utf8");
+
+    return { te, ee, heavy, stdout };
+  } finally {
+    fs.rmSync(tmpDir, { recursive: true, force: true });
+  }
+}
+
+describe("shard-split.js", () => {
+  describe("round-robin fallback (no timing data)", () => {
+    it("distributes packages evenly across shards", () => {
+      const packages = [
+        "github.com/mattermost/mattermost/server/v8/channels/api4",
+        "github.com/mattermost/mattermost/server/v8/channels/app",
+        "github.com/mattermost/mattermost/server/v8/channels/store/sqlstore",
+        "github.com/mattermost/mattermost/server/v8/config",
+      ];
+
+      // Collect assignments from all shards
+      const allTe = [];
+      for (let i = 0; i < 2; i++) {
+        const result = runSolver({ packages, shardIndex: i, shardTotal: 2 });
+        allTe.push(...result.te.split(" ").filter(Boolean));
+      }
+
+      // All packages should be assigned exactly once
+      assert.equal(allTe.sort().join("\n"), packages.sort().join("\n"));
+    });
+
+    it("uses round-robin when no timing files exist", () => {
+      const packages = ["pkg/a", "pkg/b", "pkg/c", "pkg/d", "pkg/e"];
+      const r0 = runSolver({ packages, shardIndex: 0, shardTotal: 2 });
+      const r1 = runSolver({ packages, shardIndex: 1, shardTotal: 2 });
+
+      assert.ok(r0.stdout.includes("round-robin"), "Should mention round-robin in output");
+      // No heavy runs
+      assert.equal(r0.heavy.trim(), "");
+      assert.equal(r1.heavy.trim(), "");
+    });
+  });
+
+  describe("timing-based balancing", () => {
+    it("balances shards using gotestsum.json timing data", () => {
+      const gotestsumJson = fs.readFileSync(
+        path.join(TESTDATA, "sample-gotestsum.json"),
+        "utf8"
+      );
+      const packages = [
+        "github.com/mattermost/mattermost/server/v8/channels/api4",
+        "github.com/mattermost/mattermost/server/v8/channels/app",
+        "github.com/mattermost/mattermost/server/v8/channels/store/sqlstore",
+        "github.com/mattermost/mattermost/server/v8/config",
+        "github.com/mattermost/mattermost/server/v8/enterprise/elasticsearch",
+        "github.com/mattermost/mattermost/server/v8/enterprise/compliance",
+        "github.com/mattermost/mattermost/server/public/model",
+      ];
+
+      // Run for all 4 shards and check that loads are somewhat balanced
+      const loads = [];
+      const allAssigned = new Set();
+
+      for (let i = 0; i < 4; i++) {
+        const result = runSolver({
+          packages,
+          shardIndex: i,
+          shardTotal: 4,
+          gotestsumJson,
+        });
+
+        // Track all assigned packages and tests
+        const tePkgs = result.te.split(" ").filter(Boolean);
+        const eePkgs = result.ee.split(" ").filter(Boolean);
+        tePkgs.forEach((p) => allAssigned.add(p));
+        eePkgs.forEach((p) => allAssigned.add(p));
+
+        // Parse heavy runs
+        if (result.heavy.trim()) {
+          result.heavy
+            .trim()
+            .split("\n")
+            .forEach((line) => {
+              const pkg = line.split(" ")[0];
+              allAssigned.add(pkg);
+            });
+        }
+      }
+
+      // Every package should be covered
+      for (const pkg of packages) {
+        assert.ok(
+          allAssigned.has(pkg),
+          `Package ${pkg} should be assigned to some shard`
+        );
+      }
+    });
+
+    it("does not produce empty shards with sample data", () => {
+      const gotestsumJson = fs.readFileSync(
+        path.join(TESTDATA, "sample-gotestsum.json"),
+        "utf8"
+      );
+      const packages = [
+        "github.com/mattermost/mattermost/server/v8/channels/api4",
+        "github.com/mattermost/mattermost/server/v8/channels/app",
+        "github.com/mattermost/mattermost/server/v8/channels/store/sqlstore",
+        "github.com/mattermost/mattermost/server/v8/config",
+      ];
+
+      for (let i = 0; i < 4; i++) {
+        const result = runSolver({
+          packages,
+          shardIndex: i,
+          shardTotal: 4,
+          gotestsumJson,
+        });
+        const hasWork =
+          result.te.trim() !== "" ||
+          result.ee.trim() !== "" ||
+          result.heavy.trim() !== "";
+        assert.ok(hasWork, `Shard ${i} should have some work assigned`);
+      }
+    });
+  });
+
+  describe("heavy package splitting", () => {
+    it("splits packages over HEAVY_MS threshold into individual tests", () => {
+      // Create timing data where api4 is very heavy (> 300s = 300000ms)
+      const lines = [];
+      // api4: 6 tests totaling 452.2s (> 300s threshold)
+      for (const [test, elapsed] of [
+        ["TestGetChannel", 145.2],
+        ["TestCreatePost", 98.1],
+        ["TestUpdateChannel", 72.5],
+        ["TestDeleteChannel", 58.3],
+        ["TestGetChannelMembers", 45.7],
+        ["TestSearchChannels", 32.4],
+      ]) {
+        lines.push(
+          JSON.stringify({
+            Time: "2025-03-20T10:00:00Z",
+            Action: "pass",
+            Package: "github.com/mattermost/mattermost/server/v8/channels/api4",
+            Test: test,
+            Elapsed: elapsed,
+          })
+        );
+      }
+      // config: 2 tests totaling 8s (< 120s, stays whole)
+      for (const [test, elapsed] of [
+        ["TestConfigStore", 5.0],
+        ["TestConfigMigrate", 3.0],
+      ]) {
+        lines.push(
+          JSON.stringify({
+            Time: "2025-03-20T10:00:00Z",
+            Action: "pass",
+            Package: "github.com/mattermost/mattermost/server/v8/config",
+            Test: test,
+            Elapsed: elapsed,
+          })
+        );
+      }
+
+      const gotestsumJson = lines.join("\n");
+      const packages = [
+        "github.com/mattermost/mattermost/server/v8/channels/api4",
+        "github.com/mattermost/mattermost/server/v8/config",
+      ];
+
+      // With 2 shards, api4 tests should be split across shards
+      let heavyFound = false;
+      const allHeavyTests = [];
+
+      for (let i = 0; i < 2; i++) {
+        const result = runSolver({
+          packages,
+          shardIndex: i,
+          shardTotal: 2,
+          gotestsumJson,
+        });
+
+        if (result.heavy.trim()) {
+          heavyFound = true;
+          // Parse heavy runs to extract test names
+          for (const line of result.heavy.trim().split("\n")) {
+            const parts = line.split(" ");
+            assert.equal(
+              parts[0],
+              "github.com/mattermost/mattermost/server/v8/channels/api4",
+              "Heavy package should be api4"
+            );
+            // Regex is like "^TestGetChannel$|^TestCreatePost$"
+            const tests = parts[1].split("|").map((r) => r.replace(/[\^$]/g, ""));
+            allHeavyTests.push(...tests);
+          }
+        }
+      }
+
+      assert.ok(heavyFound, "Should have heavy package splits for api4");
+      // All api4 tests should be distributed
+      const expectedTests = [
+        "TestGetChannel",
+        "TestCreatePost",
+        "TestUpdateChannel",
+        "TestDeleteChannel",
+        "TestGetChannelMembers",
+        "TestSearchChannels",
+      ];
+      assert.deepEqual(
+        allHeavyTests.sort(),
+        expectedTests.sort(),
+        "All api4 tests should be distributed across shards"
+      );
+    });
+
+    it("keeps light packages whole even with timing data", () => {
+      const gotestsumJson = [
+        '{"Action":"pass","Package":"pkg/light","Test":"TestA","Elapsed":5.0}',
+        '{"Action":"pass","Package":"pkg/light","Test":"TestB","Elapsed":3.0}',
+      ].join("\n");
+
+      const result = runSolver({
+        packages: ["pkg/light"],
+        shardIndex: 0,
+        shardTotal: 2,
+        gotestsumJson,
+      });
+
+      // Light package should be assigned whole, not split
+      assert.equal(result.heavy.trim(), "", "Light package should not be in heavy runs");
+      assert.ok(
+        result.te.includes("pkg/light"),
+        "Light package should be in TE packages"
+      );
+    });
+  });
+
+  describe("JUnit XML fallback", () => {
+    it("uses JUnit XML when gotestsum.json is missing", () => {
+      const prevReportXml = `<?xml version="1.0" encoding="UTF-8"?>
+<testsuites>
+  <testsuite name="pkg/fast" time="10.0" tests="5">
+    <testcase name="TestA" time="5.0"/>
+    <testcase name="TestB" time="5.0"/>
+  </testsuite>
+  <testsuite name="pkg/slow" time="50.0" tests="3">
+    <testcase name="TestX" time="25.0"/>
+    <testcase name="TestY" time="25.0"/>
+  </testsuite>
+</testsuites>`;
+
+      const result = runSolver({
+        packages: ["pkg/fast", "pkg/slow"],
+        shardIndex: 0,
+        shardTotal: 2,
+        prevReportXml,
+      });
+
+      assert.ok(
+        result.stdout.includes("JUnit XML"),
+        "Should indicate using JUnit XML fallback"
+      );
+      // No heavy splits with XML-only data (no per-test timing)
+      assert.equal(result.heavy.trim(), "", "Should not split packages without per-test timing");
+    });
+  });
+
+  describe("enterprise package separation", () => {
+    it("separates enterprise packages into EE output", () => {
+      const packages = [
+        "github.com/mattermost/mattermost/server/v8/channels/app",
+        "github.com/mattermost/mattermost/server/v8/enterprise/compliance",
+      ];
+
+      const result = runSolver({
+        packages,
+        shardIndex: 0,
+        shardTotal: 1,
+      });
+
+      assert.ok(
+        result.te.includes("channels/app"),
+        "TE should include non-enterprise packages"
+      );
+      assert.ok(
+        result.ee.includes("enterprise/compliance"),
+        "EE should include enterprise packages"
+      );
+      assert.ok(
+        !result.te.includes("enterprise"),
+        "TE should not include enterprise packages"
+      );
+    });
+  });
+});

server/scripts/testdata/sample-gotestsum.json

@@ -0,0 +1,22 @@
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/api4","Test":"TestGetChannel","Elapsed":45.2}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/api4","Test":"TestCreatePost","Elapsed":38.1}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/api4","Test":"TestUpdateChannel","Elapsed":22.5}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/api4","Test":"TestDeleteChannel","Elapsed":18.3}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/api4","Test":"TestGetChannelMembers","Elapsed":15.7}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/api4","Test":"TestSearchChannels","Elapsed":12.4}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/app","Test":"TestCreateUser","Elapsed":25.6}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/app","Test":"TestUpdateUser","Elapsed":20.3}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/app","Test":"TestDeleteUser","Elapsed":15.8}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/app","Test":"TestGetUser","Elapsed":10.2}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/app","Test":"TestSearchUsers","Elapsed":8.5}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/store/sqlstore","Test":"TestChannelStore","Elapsed":35.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/store/sqlstore","Test":"TestPostStore","Elapsed":28.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/channels/store/sqlstore","Test":"TestUserStore","Elapsed":22.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/config","Test":"TestConfigStore","Elapsed":5.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/config","Test":"TestConfigMigrate","Elapsed":3.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/enterprise/elasticsearch","Test":"TestSearchPosts","Elapsed":8.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/enterprise/elasticsearch","Test":"TestIndexPosts","Elapsed":6.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/v8/enterprise/compliance","Test":"TestExportCompliance","Elapsed":4.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/public/model","Test":"TestModelValidation","Elapsed":2.0}
+{"Time":"2025-03-20T10:00:00Z","Action":"pass","Package":"github.com/mattermost/mattermost/server/public/model","Test":"TestModelSerialization","Elapsed":1.5}
+{"Time":"2025-03-20T10:00:00Z","Action":"output","Package":"github.com/mattermost/mattermost/server/v8/channels/api4","Test":"TestGetChannel","Output":"--- PASS: TestGetChannel (45.20s)\n"}