In this PR we spin off usage of recently introduced packageManager-property across CI workflows and consolidate the version of pnpm/action-setup action we are using.
* Use Playwright's last info run
* Add changelog
* Fix workflow syntax
* Fix workflow syntax
* Try with continue-on-error
* Try with merge-multiple
* Try with pattern instead of name
* Print the run command
* Print the run command
* Use LAST_RUN_FLAG
* Update outputDir
* Update upload paths and parse the file content
* Don't try to download lat run info if the test type is not e2e
* Rename the downloaded file
* Rename the downloaded file
* Rename the downloaded file
* Full path to the downloaded file
* Check download path content
* Check download path content
* Check the last-run file in the right path
* Fail different tests on different run attempts
* Fix artifact name
* Force test failures
* Cleanup console logs
* Cleanup console logs
* Cleanup console logs
* Log if last run info exists but no tests are found
* Revert test changes
In this PR, we use shallow checkout if the environment provides no base_ref, which is needed for identifying changed files. If no ref provided, we are not able to calculate the diff and therefore go with faster shallow copy.
The PR scope was reduced to filtering out the `.github` folder from target changes, except the workflow itself. It should reduce the amount of the job runs when we are working with GitHub actions related files.
In this PR, we are improving the job execution time by leveraging available cached resources, refining the build artifacts cleaning approach between calculating asset sizes.
In this PR, we are consolidating babel-loader caching and adding Jest caching, improving build/testing times locally and in CI. The improvement comes from cache covering transpile steps in the mentioned processes.
* Update PULL_REQUEST_TEMPLATE.md to include a checklist item for adding/updating tests
Added a new checklist item to the pull request template for contributors to confirm they have added or updated relevant tests (E2E, Unit, Integration, etc.) to ensure their fix or feature works as intended. This helps maintain the quality and reliability of the codebase by encouraging thorough testing.
* Update workflow to request reviewers to check test coverage along with testing instructions
Modified the review-testing-instructions workflow to enhance the review process by asking reviewers to not only review the testing instructions but also verify that relevant tests (E2E, Unit, Integration, etc.) have been added or updated as needed. This change ensures a more thorough review process and helps maintain high code quality.
* Update action name
* Remove changes from Template
This PR is the second attempt to speed up monorepo installation by removing autoloading optimization setting from the configuration and performing the optimization where it is appropriate (composer post-install hooks and when building releases).
In the installation context, the composer post-install hook will execute a background task to dump the optimized autoloading, while the installation process completes other tasks.
In this PR we ar relocating playwright downloads caching into the monorepo action (behind input parameter). That consolidates monorepo-related caching and also reduces the time fetching the cache action in external workflows.
This PR drops composer autoloader optimization for the WooCommerce plugin to speed up the build process. The autoloader still gets optimized in `plugins/woocommerce/bin/build-zip.sh` file.
* Tooling: cache pnp deps
* Tooling: try fixing cache errors
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: try to parallelize js builds a bit
* Tooling: added pre-build steps to solve concurrency issues
* Tooling: limit workers
* Tooling: revert to original build command
* Tooling: playwright related tweaks
* Tooling: minor tweaks.
* Tooling: fix caching path.
* Tooling: trigger php builds to test deps caching.
* Tooling: trigger php builds to test deps caching.
* Tooling: lock caching to monorepo setup, so it doesn't slow down build steps.
* Tooling: lock Playwright cache to build, as build is pre-requisite for running tests.
* Tooling: added caching build-folders.
* Tooling: added caching build-folders.
* Tooling: added caching build-folders.
* Revert "Tooling: added caching build-folders."
This reverts commit c57b9dff13.
* Tooling: added caching build-folders.
* Tooling: cleanup sweep.
* Tooling: cleanup sweep.
* Tooling: added a changelog entry.
* Tooling: cleanup.
* Tooling: try speedup live branches build.
* Tooling: try speedup live branches build.
* Tooling: cleanup.
* Tooling: try speedup live branches build.
* Tooling: try speedup live branches build.
* * Use run_attempt in the artifact name
* Remove the Allure report steps
* Test change to trigger tests
* Only run the publish report step if there is an artifact-id
* Only run the publish report step if there is an artifact-id
* Revert test changes
* Also condition the deprecated publish report step
* Update issue template to use the new SSR feature for GitHub
* Update issue template to use the new SSR feature for GitHub
* Update .github/ISSUE_TEMPLATE/1-bug-report.yml
---------
Co-authored-by: Corey McKrill <916023+coreymckrill@users.noreply.github.com>
* Fix api tests results path
* Add changelog
* Run Core API tests - HPOS disabled on pull_request
* Update the artifacts name pattern to avoid conflicts
* Revert pull_request trigger
* Add a call to a new report workflow in woocommerce-test-reports repo
* Test change to trigger tests
* Update report title
* Test change to trigger tests
* Check REPORT_TITLE
* Add quotes on all arguments
* Update reporting configuration to all core e2e jobs
* Add changelog
* Use GITHUB_HEAD_REF for pull_request
* Set REPORT_TITLE for workflow dispatch
* Updated trigger from daily-e2e to daily-checks
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
* Fix the report name sent to woocommerce-test-reports workflow
* Add test type
* Test change
* Remove test type
* Merge artifacts to keep allure-results
* Overwrite artifacts if one already exists with the same name
* Test changes to trigger tests
* Fix allure-results and .state paths
* Filter tests for faster feedback
* Debug artifacts content
* Use name instead of pattern
* Use name instead of pattern
* Don't use an intermediate path
* Use artifact name and path instead of only pattern
* Use artifact name and path instead of only pattern
* Use artifact name and path instead of only pattern
* Remove testMatch filter to run all tests
* Revert test change
* Add changelog
* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Add release checks jobs and deprecate smoke-test-release.yml
* Deprecate smoke-test-release.yml
* Add tests-release-checks.yml to trigger ci.yml on push to tags
* Add the pull_request trigger to test the workflow
* Remove the pull_request trigger to test the workflow
* Add the pull_request trigger to test the workflow
* Add the tag input in ci.yml
* Remove import and use core reference
* Only build if build-plugin is true
* Use release-artifact instead
* Split setup-monorepo step
* Use tag: 'nightly' to test
* Use inputs.tag directly in the wp-env setup step
* Add script to override the wp-env config
* Run the env config script in ci
* Set the artifact name
* Fix workflow syntax
* Add changelog
* Validate RELEASE_TAG and ARTIFACT_NAME
* Use real ref type and name
* Use real ref type and name
* Use real ref type and name
* Use github context instead
* Test with 9.0.0-beta.2
* Add push to tags trigger directly in ci.yml
* Configure one job to run with nightly-checks
* Remove empty lines
* Format events array one element per line
* Format events array one element per line
* Add flaky test reporting and issue creation for Blocks
This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow.
- The reporter captures flaky test results and saves them to `flaky-tests` directory.
- If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated.
- The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues.
1. **Workflow Changes**:
- **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs.
- **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards.
- **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present.
2. **New Flaky Test Reporter**:
- Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests.
3. **Playwright Configuration**:
- Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI.
4. **Demo Tests**:
- Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality.
Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites.
* Remove draft condition for testing
* 'Merge failures artifacts' job shouldn't touch flaky test artifacts
* Try fixing if condition
* Add required permissions
* Let's try providing all the permissions
* Let's try providing all the permissions
* Create flaky tests issues when PR is merged with trunk
* Update if condition
* Run action for specific paths only
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Remove Demo file
* Remove Demo file
* Use commit hash to use exact version of the script
* Replace label with team: Kirigami & Origami
---------
Co-authored-by: github-actions <github-actions@github.com>
* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Update PR template and auto change log
* Ignore .github md files in linting
* Rebuilt monorepo-utils to solve conflict
* Updated regex pattern to include uppercase X
* Move the changelog entry not required checkbox on top of the description.
* Return the comment even if shouldAutomateNoChangelog is false
* Updated tests
* Rebuilt monorepo-utils
---------
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
* Add the slack-test-report util
* Error if the required GitHub context variables are not set
* Error if the required SLACK_CHANNEL env var is not set
* Use a more generic message to be able to report on jobs that are not tests, like build
* Add the alert-on-failure job in ci
* Run for pull_request to test the job
* Setup monorepo first
* Force a linting error
* Better logging
* Use inputs.trigger as report name if set
* Force an e2e test failure
* Set the commit message in the notification
* Use INPUT_TRIGGER for all notifications
* Revert test changes
* Add condition on github.event_name for lint jobs
* Update a file to trigger a lint job
* Update condition to run only on pull_request
* Revert test change
* Always run the Slack notification step
* Add pull_request test trigger
* Update report step
* Remove branch filter
* Try with push event and branch filter
* Pass secrets to ci.yml
* Update testIgnore pattern for Gutenberg project
* Add changelog
* Remove the test push trigger
* One more test run
* Removed push test trigger
* Add workflow_call event
* Add tests-daily-run.yml
* Remove workflow version
* Update input value
* Remove secrets from caller workflow
* Rename job
* Check context
* Check context
* Check workflow_call inputs
* Override github.event_name with inputs.trigger
* Fix input name
* Add config for daily e2e tests with Gutenberg
* Update env commands for Gutenberg run
* Add disableHpos env variable in ci-utils
* Revert the trigger to push
* Revert the env variable setting in ci-jobs
* Revert the env variable setting in ci-jobs
* Rebuild utils
* Update test env start command
* Define the Gutenberg Playwright project
* Use path relative to workspace root
* Update events for test projects
* Update events for test projects
* Add changelogs
* Remove the e2e tests with Gutenberg tests from daily runs
* Set the schedule and workflow_dispatch triggers
* Add --project default flag to only run the default tests in CI
* Renamed snapshots to match the new Playwright project name
* Add disableHpos test env variable
* Update events for HPOS disabled tests
* Add pull_request event to test the env
* Remove the pull_request event
* Add changelog
* Make the disableHpos var a boolean
* Also set the env in ci.yml
* Remove console log statement
* Print woocommerce_custom_orders_table_enabled
* Run test setup script again to check output
* Capture env setup output and print it
* Get woocommerce_custom_orders_table_enabled value in test env script
* Get more details on HPOS setting
* Remove the DISABLE_HPOS logic from test env setup script
* Use DISABLE_HPOS instead of ENABLE_HPOS everywhere
* Turn off no-console and jest/no-standalone-expect eslint rules for api core tests
* Test API tests with HPOS disabled
* Fix eslint error
* Clean up debug statements
* Remove pull_request test trigger
* Include performance tests job in the needed jobs for evaluation
* Use Github api to get the workflow jobs status
* Evaluate project-jobs
* See results
* Check for cancelled jobs
* Add evaluate-project-jobs.js and rename optional jobs
* Fix script path
* Checkout first
* And then fix the path again.
* Update env variable name and be more detailed in what are the required variables
* Use a flat matrix env variable
* Update to trigger test jobs
* Use the jobs name to evaluate optional requirement
* Fix conditions
* Prettier print
* Test lint job failure
* Remove unused MATRIX env variable
* Fix test
* Remove unused MATRIX variable check
* Only run my account e2e tests
* Remove unused function
* Nicer console printing
* Revert change that triggers failing lint job
* Force failing e2e test
* Revert e2e test command and forced failure
* Added test data and test mode
* Added more test data
* Fixed lint errors and warnings
* Exclude .github folder from eslintignore
* Change to trigger everything
* Revert change to trigger everything
* try db reset in page teardown
* move reset to setup step
* Use wp db cli
* Fix global setup
That part is overriding the logged in user state and wipes out the nonce and rootUrl fields.
* Try importing instead from a generated dump
* Revert "Try importing instead from a generated dump"
This reverts commit 987dc471c9.
* Revert "Revert "Try importing instead from a generated dump""
This reverts commit c8d008cb20.
* Don't bypass visitSiteEditor so that the Welcome Guide is closed
* use createNewPost
* Revert "Revert "Revert "Try importing instead from a generated dump"""
This reverts commit 2684273582.
* [Blocks]: Fix E2E tests (#46344)
* Load local pickup enabled setting as bool not string
* Add changelog
* Remove unused import
* Update plugins/woocommerce-blocks/assets/js/extensions/shipping-methods/pickup-location/utils.ts
Co-authored-by: Niels Lange <info@nielslange.de>
* Use strict equality test
* try now
* try now
* add a new sql file
* remove default.sql
* fix welcome guide tour
* fix E2E tests
* Fix the template revert tests
...where the template is unreachable due to pagination.
* Add changelog entry
* improve logic to closeWelcomeGuideModal
* fix cart checkout tests
* improve flakiness
* improve flakiness
* fix flaky test
* fix company field
* fix E2E test
* fix E2E tests
---------
Co-authored-by: Thomas Roberts <thomas.roberts@automattic.com>
Co-authored-by: Thomas Roberts <5656702+opr@users.noreply.github.com>
Co-authored-by: Niels Lange <info@nielslange.de>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
* Replace beforeAll w/ beforeEach + remove all after* hooks
* Fix broken hooks
* Activate plugins via requestUtils API
* [Blocks - E2E]: Add `playwright/no-hooks` ESlint rule (#46432)
add ESLint configuration
* Clean up console logs
* Remove obsolete language setup steps
* Remove more unnecessary setup steps
* Remove even more obsolete setup steps
* tmp: add the LYS fix
* Try stabilizing the company field test
* Use canvas param instead of manually entering edit mode
* Remove double site editor redirect
* Revert "Use canvas param instead of manually entering edit mode"
This reverts commit 5e6cc17154.
* Revert "Remove double site editor redirect"
This reverts commit 69a57a82a8.
* Fix flaky products sorting test
* Blocks: Fix ESLint errors (#46595)
fix eslint error
* Fix ESLint errors (#46626)
* Fix ESLint errors
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
---------
Co-authored-by: github-actions <github-actions@github.com>
* Blocks: Migrate multiple tests to Playwright
* fix path
* migrate other tests
* add product best sellers E2E tests
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* migrate jest-puppetter E2E tests to Playwright
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Blocks: Delete Jest-Puppeteer infrastructure
* Add changefile(s) from automation for the following project(s): woocommerce-blocks, woocommerce
* Remove obsolete waitForLoadState call
* improve E2E tests
* fix E2E tests
* improve E2E tests
* remove old test
* remove not necessary test
* fix number product
* fix E2E test
* fix number of products
* fix product count
* refactor store notice E2E tests
* update string
* fix E2E test
---------
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Thomas Roberts <thomas.roberts@automattic.com>
Co-authored-by: Thomas Roberts <5656702+opr@users.noreply.github.com>
Co-authored-by: Niels Lange <info@nielslange.de>
Co-authored-by: github-actions <github-actions@github.com>
Adrian Moldovan
louwie17
Bart Kalisz
Chi-Hsuan Huang
Jason Kytros
Vladimir Reznichenko
Jonathan Lane
Manish Menaria
jonathansadowski
Naman Malhotra
Barry Hughes
Ron Rennick
Sam Seay
Moon
Luigi Teschio