* Add flaky test reporting and issue creation for Blocks
This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow.
- The reporter captures flaky test results and saves them to `flaky-tests` directory.
- If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated.
- The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues.
1. **Workflow Changes**:
- **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs.
- **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards.
- **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present.
2. **New Flaky Test Reporter**:
- Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests.
3. **Playwright Configuration**:
- Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI.
4. **Demo Tests**:
- Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality.
Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites.
* Remove draft condition for testing
* 'Merge failures artifacts' job shouldn't touch flaky test artifacts
* Try fixing if condition
* Add required permissions
* Let's try providing all the permissions
* Let's try providing all the permissions
* Create flaky tests issues when PR is merged with trunk
* Update if condition
* Run action for specific paths only
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Remove Demo file
* Remove Demo file
* Use commit hash to use exact version of the script
* Replace label with team: Kirigami & Origami
---------
Co-authored-by: github-actions <github-actions@github.com>
Karol Manijak
Adrian Moldovan
Manish Menaria