* Add flaky test reporting and issue creation for Blocks
This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow.
- The reporter captures flaky test results and saves them to `flaky-tests` directory.
- If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated.
- The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues.
1. **Workflow Changes**:
- **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs.
- **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards.
- **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present.
2. **New Flaky Test Reporter**:
- Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests.
3. **Playwright Configuration**:
- Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI.
4. **Demo Tests**:
- Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality.
Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites.
* Remove draft condition for testing
* 'Merge failures artifacts' job shouldn't touch flaky test artifacts
* Try fixing if condition
* Add required permissions
* Let's try providing all the permissions
* Let's try providing all the permissions
* Create flaky tests issues when PR is merged with trunk
* Update if condition
* Run action for specific paths only
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Remove Demo file
* Remove Demo file
* Use commit hash to use exact version of the script
* Replace label with team: Kirigami & Origami
---------
Co-authored-by: github-actions <github-actions@github.com>
* Revert "[e2e tests blocks] Configure the number of shards for each matrix project (#45212)"
This reverts commit 72789656cf.
* trigger CI
* run E2E tests
* restore ci.yml file
* trigger E2E tests
* disable E2E tests about mini cart translations
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Fix the setupAndFetchQueryContextURL utility
* add comment
* Remove obsolete test
* Blocks E2E: Update timeout values (#45353)
Set timeout values same as core
---------
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
* Add sharding to Blocks e2e tests
* Changelog
* Push a change to Blocks README to trigger tests
* Fix syntax
* Fixed again
* Try again
* Trying to get the shard number
* Shard names
* fix E2E test
* complete all the jobs
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Luigi Teschio <gigitux@gmail.com>
* Update commands to run e2e tests
* Capitalize Product Catalog template name
* Create tests with block theme with templates
* Create Block Theme with Templates test environment
* Update classic theme commands
* Code cleanup
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Code cleanup (II)
* Add config to blocks-playwright.yml and remove unnecesssary .github/workflows/playwright.yml file
* Remove unnecessary imports from product-catalog-template.block_theme_with_templates.spec.ts
* Update blocks e2e commands to pnpm
* Replace hardcoded strings with a variable
* Unify all 'block theme with templates' tests in a parameterized test
* Create a testData object for the Mini-Cart template part test
* Fix some code styling issues
* Create a testData object for the Mini-Cart template part test (II)
* Remove duplicate userText variable
---------
Co-authored-by: github-actions <github-actions@github.com>
* First pass at Playwright test migration
* Set working directory
* Remove exec
* Configure the environment before testing
* Remove filter
* Update paths for tests
* Try explicitly stating working-directory
* Change tests directory
* Try running all tests
* Uncommented project
* Update commands to run tests
* Comment out env config
* Uncomment
* Changelog
* Set up multiple test configurations
* Update failing test
* Update paths for artifacts
* Update path to artifacts
* Path to results
* remove unnecessary file
* Skip translation E2E tests
Skipping translation tests until we fix the missing translations issues after we changed the domain to "woocommerce"
* Update the artifacts path
* Fix lint errors
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Luigi Teschio <gigitux@gmail.com>
Co-authored-by: Saad Tarhi <saad.trh@gmail.com>
Manish Menaria
Bart Kalisz
Luigi Teschio
Adrian Moldovan
Jonathan Lane
Albert Juhé Lluveras