In this PR, we are consolidating babel-loader caching and adding Jest caching, improving build/testing times locally and in CI. The improvement comes from cache covering transpile steps in the mentioned processes.
* Update PULL_REQUEST_TEMPLATE.md to include a checklist item for adding/updating tests
Added a new checklist item to the pull request template for contributors to confirm they have added or updated relevant tests (E2E, Unit, Integration, etc.) to ensure their fix or feature works as intended. This helps maintain the quality and reliability of the codebase by encouraging thorough testing.
* Update workflow to request reviewers to check test coverage along with testing instructions
Modified the review-testing-instructions workflow to enhance the review process by asking reviewers to not only review the testing instructions but also verify that relevant tests (E2E, Unit, Integration, etc.) have been added or updated as needed. This change ensures a more thorough review process and helps maintain high code quality.
* Update action name
* Remove changes from Template
This PR is the second attempt to speed up monorepo installation by removing autoloading optimization setting from the configuration and performing the optimization where it is appropriate (composer post-install hooks and when building releases).
In the installation context, the composer post-install hook will execute a background task to dump the optimized autoloading, while the installation process completes other tasks.
In this PR we ar relocating playwright downloads caching into the monorepo action (behind input parameter). That consolidates monorepo-related caching and also reduces the time fetching the cache action in external workflows.
This PR drops composer autoloader optimization for the WooCommerce plugin to speed up the build process. The autoloader still gets optimized in `plugins/woocommerce/bin/build-zip.sh` file.
* Tooling: cache pnp deps
* Tooling: try fixing cache errors
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: added pre-build steps to solve concurrency issues
* Tooling: limit workers
* Tooling: revert to original build command
* Tooling: playwright related tweaks
* Tooling: minor tweaks.
* Tooling: fix caching path.
* Tooling: trigger php builds to test deps caching.
* Tooling: trigger php builds to test deps caching.
* Tooling: lock caching to monorepo setup, so it doesn't slow down build steps.
* Tooling: lock Playwright cache to build, as build is pre-requisite for running tests.
* Tooling: added caching build-folders.
* Tooling: added caching build-folders.
* Tooling: added caching build-folders.
* Revert "Tooling: added caching build-folders."
This reverts commit c57b9dff13.
* Tooling: added caching build-folders.
* Tooling: cleanup sweep.
* Tooling: cleanup sweep.
* Tooling: added a changelog entry.
* Tooling: cleanup.
* Tooling: try speedup live branches build.
* Tooling: try speedup live branches build.
* Tooling: cleanup.
* Tooling: try speedup live branches build.
* Tooling: try speedup live branches build.
* * Use run_attempt in the artifact name
* Remove the Allure report steps
* Test change to trigger tests
* Only run the publish report step if there is an artifact-id
* Only run the publish report step if there is an artifact-id
* Revert test changes
* Also condition the deprecated publish report step
* Update issue template to use the new SSR feature for GitHub
* Update issue template to use the new SSR feature for GitHub
* Update .github/ISSUE_TEMPLATE/1-bug-report.yml
---------
Co-authored-by: Corey McKrill <916023+coreymckrill@users.noreply.github.com>
* Fix api tests results path
* Add changelog
* Run Core API tests - HPOS disabled on pull_request
* Update the artifacts name pattern to avoid conflicts
* Revert pull_request trigger
* Add a call to a new report workflow in woocommerce-test-reports repo
* Test change to trigger tests
* Update report title
* Test change to trigger tests
* Check REPORT_TITLE
* Add quotes on all arguments
* Update reporting configuration to all core e2e jobs
* Add changelog
* Use GITHUB_HEAD_REF for pull_request
* Set REPORT_TITLE for workflow dispatch
* Updated trigger from daily-e2e to daily-checks
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
* Fix the report name sent to woocommerce-test-reports workflow
* Add test type
* Test change
* Remove test type
* Merge artifacts to keep allure-results
* Overwrite artifacts if one already exists with the same name
* Test changes to trigger tests
* Fix allure-results and .state paths
* Filter tests for faster feedback
* Debug artifacts content
* Use name instead of pattern
* Use name instead of pattern
* Don't use an intermediate path
* Use artifact name and path instead of only pattern
* Use artifact name and path instead of only pattern
* Use artifact name and path instead of only pattern
* Remove testMatch filter to run all tests
* Revert test change
* Add changelog
* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Add release checks jobs and deprecate smoke-test-release.yml
* Deprecate smoke-test-release.yml
* Add tests-release-checks.yml to trigger ci.yml on push to tags
* Add the pull_request trigger to test the workflow
* Remove the pull_request trigger to test the workflow
* Add the pull_request trigger to test the workflow
* Add the tag input in ci.yml
* Remove import and use core reference
* Only build if build-plugin is true
* Use release-artifact instead
* Split setup-monorepo step
* Use tag: 'nightly' to test
* Use inputs.tag directly in the wp-env setup step
* Add script to override the wp-env config
* Run the env config script in ci
* Set the artifact name
* Fix workflow syntax
* Add changelog
* Validate RELEASE_TAG and ARTIFACT_NAME
* Use real ref type and name
* Use real ref type and name
* Use real ref type and name
* Use github context instead
* Test with 9.0.0-beta.2
* Add push to tags trigger directly in ci.yml
* Configure one job to run with nightly-checks
* Remove empty lines
* Format events array one element per line
* Format events array one element per line
* Add flaky test reporting and issue creation for Blocks
This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow.
- The reporter captures flaky test results and saves them to `flaky-tests` directory.
- If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated.
- The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues.
1. **Workflow Changes**:
- **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs.
- **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards.
- **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present.
2. **New Flaky Test Reporter**:
- Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests.
3. **Playwright Configuration**:
- Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI.
4. **Demo Tests**:
- Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality.
Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites.
* Remove draft condition for testing
* 'Merge failures artifacts' job shouldn't touch flaky test artifacts
* Try fixing if condition
* Add required permissions
* Let's try providing all the permissions
* Let's try providing all the permissions
* Create flaky tests issues when PR is merged with trunk
* Update if condition
* Run action for specific paths only
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Remove Demo file
* Remove Demo file
* Use commit hash to use exact version of the script
* Replace label with team: Kirigami & Origami
---------
Co-authored-by: github-actions <github-actions@github.com>
* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Update PR template and auto change log
* Ignore .github md files in linting
* Rebuilt monorepo-utils to solve conflict
* Updated regex pattern to include uppercase X
* Move the changelog entry not required checkbox on top of the description.
* Return the comment even if shouldAutomateNoChangelog is false
* Updated tests
* Rebuilt monorepo-utils
---------
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
Adrian Moldovan
Vladimir Reznichenko
Manish Menaria
jonathansadowski
Naman Malhotra
Barry Hughes
Jonathan Lane
Ron Rennick
Sam Seay