woocommerce/.github/workflows/deprecated/blocks-playwright.yml

147 lines
5.4 KiB
YAML
Raw Normal View History

name: Blocks Playwright Tests
on:
pull_request:
paths:
- '.github/workflows/blocks-playwright.yml' # This file
- 'plugins/woocommerce-blocks/**'
- 'plugins/woocommerce/src/Blocks/**'
- 'plugins/woocommerce/templates/**'
- 'plugins/woocommerce/patterns/**'
# Allow manually triggering the workflow.
workflow_dispatch:
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
# Run workflow when a PR is merged to trunk branch
# to create github issues for flaky tests.
push:
branches:
- trunk
paths:
- '.github/workflows/blocks-playwright.yml' # This file
- 'plugins/woocommerce-blocks/**'
- 'plugins/woocommerce/src/Blocks/**'
- 'plugins/woocommerce/templates/**'
- 'plugins/woocommerce/patterns/**'
concurrency:
group: '${{ github.workflow }}-${{ github.ref }}'
cancel-in-progress: true
env:
FORCE_COLOR: 1
jobs:
blocks-playwright-tests:
name: Shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }}
timeout-minutes: 60
runs-on: ubuntu-latest
defaults:
run:
working-directory: plugins/woocommerce-blocks
strategy:
fail-fast: false
matrix:
shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
shardTotal: [10]
steps:
- uses: actions/checkout@v4
- name: Setup WooCommerce Monorepo
uses: ./.github/actions/setup-woocommerce-monorepo
with:
install: '@woocommerce/plugin-woocommerce...'
build: '@woocommerce/plugin-woocommerce'
pull-playwright-cache: true
pull-package-deps: '@woocommerce/plugin-woocommerce'
- name: Install Playwright dependencies
run: pnpm exec playwright install chromium --with-deps
- name: Setup testing environment and start the server
run: pnpm env:start
- name: Run Playwright tests
run: pnpm test:e2e --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
- name: Archive debug artifacts (screenshots, traces)
uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: failures-artifacts-shard-${{ matrix.shardIndex }}
path: plugins/woocommerce-blocks/tests/e2e/artifacts/test-results
if-no-files-found: ignore
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
- name: Archive flaky test reports
uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: flaky-tests-report-shard-${{ matrix.shardIndex }}
path: plugins/woocommerce-blocks/flaky-tests
if-no-files-found: ignore
merge-artifacts:
# Merges all artifacts from all shards into a single zip and
# deletes the parts. In case of a rerun, artifacts from the
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
# previous run will be retained by merging them with the new ones.
name: Merge Artifacts
if: ${{ !cancelled() }}
needs: [blocks-playwright-tests]
runs-on: ubuntu-latest
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
outputs:
has-flaky-test-report: ${{ !!steps.merge-flaky-tests-reports.outputs.artifact-id }}
steps:
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
- name: Merge failures artifacts
uses: actions/upload-artifact/merge@v4
# Don't fail the job if there aren't any artifacts to merge.
continue-on-error: true
with:
name: failures-artifacts
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
pattern: failures-artifacts*
delete-merged: true
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
- name: Merge flaky tests reports
id: merge-flaky-tests-reports
uses: actions/upload-artifact/merge@v4
continue-on-error: true
with:
name: flaky-tests-report
pattern: flaky-tests-report*
delete-merged: true
create-github-issues-for-flaky-tests:
name: Create GitHub issues for flaky tests
needs: [merge-artifacts]
if: ${{ !cancelled() && needs.merge-artifacts.outputs.has-flaky-test-report == 'true' }}
runs-on: ubuntu-latest
permissions:
contents: read
issues: write
steps:
# We are using @wordpress/report-flaky-tests package from Gutenberg repo
# to create GitHub issues for flaky tests.
- uses: actions/checkout@v4
with:
repository: WordPress/gutenberg
# We are using commit hash to checkout the exact version of the script.
# This is to avoid any breaking changes.
ref: dbf201449e9736f672b61e422787d47659db327a
- uses: actions/download-artifact@v4
with:
name: flaky-tests-report
path: flaky-tests
- name: Setup Node.js and install dependencies
uses: ./.github/setup-node
- name: Npm build
# TODO: We don't have to build the entire project, just the action itself.
run: npm run build:packages
- name: Report flaky tests
uses: ./packages/report-flaky-tests
with:
repo-token: '${{ secrets.GITHUB_TOKEN }}'
label: 'metric: flaky e2e test'
WooCommerce Blocks: Added a GitHub Action to create issues for flaky E2E tests (#47758) * Add flaky test reporting and issue creation for Blocks This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow. - The reporter captures flaky test results and saves them to `flaky-tests` directory. - If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated. - The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues. 1. **Workflow Changes**: - **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs. - **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards. - **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present. 2. **New Flaky Test Reporter**: - Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests. 3. **Playwright Configuration**: - Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI. 4. **Demo Tests**: - Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality. Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites. * Remove draft condition for testing * 'Merge failures artifacts' job shouldn't touch flaky test artifacts * Try fixing if condition * Add required permissions * Let's try providing all the permissions * Let's try providing all the permissions * Create flaky tests issues when PR is merged with trunk * Update if condition * Run action for specific paths only * Add changefile(s) from automation for the following project(s): woocommerce-blocks * Remove Demo file * Remove Demo file * Use commit hash to use exact version of the script * Replace label with team: Kirigami & Origami --------- Co-authored-by: github-actions <github-actions@github.com>
2024-06-05 14:19:28 +00:00
artifact-path: flaky-tests