* Fix the report name sent to woocommerce-test-reports workflow
* Add test type
* Test change
* Remove test type
* Merge artifacts to keep allure-results
* Overwrite artifacts if one already exists with the same name
* Test changes to trigger tests
* Fix allure-results and .state paths
* Filter tests for faster feedback
* Debug artifacts content
* Use name instead of pattern
* Use name instead of pattern
* Don't use an intermediate path
* Use artifact name and path instead of only pattern
* Use artifact name and path instead of only pattern
* Use artifact name and path instead of only pattern
* Remove testMatch filter to run all tests
* Revert test change
* Add changelog
* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Add release checks jobs and deprecate smoke-test-release.yml
* Deprecate smoke-test-release.yml
* Add tests-release-checks.yml to trigger ci.yml on push to tags
* Add the pull_request trigger to test the workflow
* Remove the pull_request trigger to test the workflow
* Add the pull_request trigger to test the workflow
* Add the tag input in ci.yml
* Remove import and use core reference
* Only build if build-plugin is true
* Use release-artifact instead
* Split setup-monorepo step
* Use tag: 'nightly' to test
* Use inputs.tag directly in the wp-env setup step
* Add script to override the wp-env config
* Run the env config script in ci
* Set the artifact name
* Fix workflow syntax
* Add changelog
* Validate RELEASE_TAG and ARTIFACT_NAME
* Use real ref type and name
* Use real ref type and name
* Use real ref type and name
* Use github context instead
* Test with 9.0.0-beta.2
* Add push to tags trigger directly in ci.yml
* Configure one job to run with nightly-checks
* Remove empty lines
* Format events array one element per line
* Format events array one element per line
* Add flaky test reporting and issue creation for Blocks
This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow.
- The reporter captures flaky test results and saves them to `flaky-tests` directory.
- If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated.
- The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues.
1. **Workflow Changes**:
- **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs.
- **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards.
- **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present.
2. **New Flaky Test Reporter**:
- Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests.
3. **Playwright Configuration**:
- Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI.
4. **Demo Tests**:
- Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality.
Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites.
* Remove draft condition for testing
* 'Merge failures artifacts' job shouldn't touch flaky test artifacts
* Try fixing if condition
* Add required permissions
* Let's try providing all the permissions
* Let's try providing all the permissions
* Create flaky tests issues when PR is merged with trunk
* Update if condition
* Run action for specific paths only
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Remove Demo file
* Remove Demo file
* Use commit hash to use exact version of the script
* Replace label with team: Kirigami & Origami
---------
Co-authored-by: github-actions <github-actions@github.com>
* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Update PR template and auto change log
* Ignore .github md files in linting
* Rebuilt monorepo-utils to solve conflict
* Updated regex pattern to include uppercase X
* Move the changelog entry not required checkbox on top of the description.
* Return the comment even if shouldAutomateNoChangelog is false
* Updated tests
* Rebuilt monorepo-utils
---------
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
* Add the slack-test-report util
* Error if the required GitHub context variables are not set
* Error if the required SLACK_CHANNEL env var is not set
* Use a more generic message to be able to report on jobs that are not tests, like build
* Add the alert-on-failure job in ci
* Run for pull_request to test the job
* Setup monorepo first
* Force a linting error
* Better logging
* Use inputs.trigger as report name if set
* Force an e2e test failure
* Set the commit message in the notification
* Use INPUT_TRIGGER for all notifications
* Revert test changes
* Add condition on github.event_name for lint jobs
* Update a file to trigger a lint job
* Update condition to run only on pull_request
* Revert test change
* Always run the Slack notification step
* Add pull_request test trigger
* Update report step
* Remove branch filter
* Try with push event and branch filter
* Pass secrets to ci.yml
* Update testIgnore pattern for Gutenberg project
* Add changelog
* Remove the test push trigger
* One more test run
* Removed push test trigger
* Add workflow_call event
* Add tests-daily-run.yml
* Remove workflow version
* Update input value
* Remove secrets from caller workflow
* Rename job
* Check context
* Check context
* Check workflow_call inputs
* Override github.event_name with inputs.trigger
* Fix input name
* Add config for daily e2e tests with Gutenberg
* Update env commands for Gutenberg run
* Add disableHpos env variable in ci-utils
* Revert the trigger to push
* Revert the env variable setting in ci-jobs
* Revert the env variable setting in ci-jobs
* Rebuild utils
* Update test env start command
* Define the Gutenberg Playwright project
* Use path relative to workspace root
* Update events for test projects
* Update events for test projects
* Add changelogs
* Remove the e2e tests with Gutenberg tests from daily runs
* Set the schedule and workflow_dispatch triggers
* Add --project default flag to only run the default tests in CI
* Renamed snapshots to match the new Playwright project name
* Add disableHpos test env variable
* Update events for HPOS disabled tests
* Add pull_request event to test the env
* Remove the pull_request event
* Add changelog
* Make the disableHpos var a boolean
* Also set the env in ci.yml
* Remove console log statement
* Print woocommerce_custom_orders_table_enabled
* Run test setup script again to check output
* Capture env setup output and print it
* Get woocommerce_custom_orders_table_enabled value in test env script
* Get more details on HPOS setting
* Remove the DISABLE_HPOS logic from test env setup script
* Use DISABLE_HPOS instead of ENABLE_HPOS everywhere
* Turn off no-console and jest/no-standalone-expect eslint rules for api core tests
* Test API tests with HPOS disabled
* Fix eslint error
* Clean up debug statements
* Remove pull_request test trigger
* Include performance tests job in the needed jobs for evaluation
* Use Github api to get the workflow jobs status
* Evaluate project-jobs
* See results
* Check for cancelled jobs
* Add evaluate-project-jobs.js and rename optional jobs
* Fix script path
* Checkout first
* And then fix the path again.
* Update env variable name and be more detailed in what are the required variables
* Use a flat matrix env variable
* Update to trigger test jobs
* Use the jobs name to evaluate optional requirement
* Fix conditions
* Prettier print
* Test lint job failure
* Remove unused MATRIX env variable
* Fix test
* Remove unused MATRIX variable check
* Only run my account e2e tests
* Remove unused function
* Nicer console printing
* Revert change that triggers failing lint job
* Force failing e2e test
* Revert e2e test command and forced failure
* Added test data and test mode
* Added more test data
* Fixed lint errors and warnings
* Exclude .github folder from eslintignore
* Change to trigger everything
* Revert change to trigger everything
* try db reset in page teardown
* move reset to setup step
* Use wp db cli
* Fix global setup
That part is overriding the logged in user state and wipes out the nonce and rootUrl fields.
* Try importing instead from a generated dump
* Revert "Try importing instead from a generated dump"
This reverts commit 987dc471c9.
* Revert "Revert "Try importing instead from a generated dump""
This reverts commit c8d008cb20.
* Don't bypass visitSiteEditor so that the Welcome Guide is closed
* use createNewPost
* Revert "Revert "Revert "Try importing instead from a generated dump"""
This reverts commit 2684273582.
* [Blocks]: Fix E2E tests (#46344)
* Load local pickup enabled setting as bool not string
* Add changelog
* Remove unused import
* Update plugins/woocommerce-blocks/assets/js/extensions/shipping-methods/pickup-location/utils.ts
Co-authored-by: Niels Lange <info@nielslange.de>
* Use strict equality test
* try now
* try now
* add a new sql file
* remove default.sql
* fix welcome guide tour
* fix E2E tests
* Fix the template revert tests
...where the template is unreachable due to pagination.
* Add changelog entry
* improve logic to closeWelcomeGuideModal
* fix cart checkout tests
* improve flakiness
* improve flakiness
* fix flaky test
* fix company field
* fix E2E test
* fix E2E tests
---------
Co-authored-by: Thomas Roberts <thomas.roberts@automattic.com>
Co-authored-by: Thomas Roberts <5656702+opr@users.noreply.github.com>
Co-authored-by: Niels Lange <info@nielslange.de>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
* Replace beforeAll w/ beforeEach + remove all after* hooks
* Fix broken hooks
* Activate plugins via requestUtils API
* [Blocks - E2E]: Add `playwright/no-hooks` ESlint rule (#46432)
add ESLint configuration
* Clean up console logs
* Remove obsolete language setup steps
* Remove more unnecessary setup steps
* Remove even more obsolete setup steps
* tmp: add the LYS fix
* Try stabilizing the company field test
* Use canvas param instead of manually entering edit mode
* Remove double site editor redirect
* Revert "Use canvas param instead of manually entering edit mode"
This reverts commit 5e6cc17154.
* Revert "Remove double site editor redirect"
This reverts commit 69a57a82a8.
* Fix flaky products sorting test
* Blocks: Fix ESLint errors (#46595)
fix eslint error
* Fix ESLint errors (#46626)
* Fix ESLint errors
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
---------
Co-authored-by: github-actions <github-actions@github.com>
* Blocks: Migrate multiple tests to Playwright
* fix path
* migrate other tests
* add product best sellers E2E tests
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* migrate jest-puppetter E2E tests to Playwright
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Blocks: Delete Jest-Puppeteer infrastructure
* Add changefile(s) from automation for the following project(s): woocommerce-blocks, woocommerce
* Remove obsolete waitForLoadState call
* improve E2E tests
* fix E2E tests
* improve E2E tests
* remove old test
* remove not necessary test
* fix number product
* fix E2E test
* fix number of products
* fix product count
* refactor store notice E2E tests
* update string
* fix E2E test
---------
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Thomas Roberts <thomas.roberts@automattic.com>
Co-authored-by: Thomas Roberts <5656702+opr@users.noreply.github.com>
Co-authored-by: Niels Lange <info@nielslange.de>
Co-authored-by: github-actions <github-actions@github.com>
* Update env setup script to disable hpos
* Use DISABLE_HPOS instead of ENABLE_HPOS
* Log HPOS state
* Configure project for core e2e with HPOS disabled
* Add event argument for ci-jobs util
* Add event option for test jobs
* Add support for github events
* Add changelog
* Configure HPOS tests to run on pull_request
* Fix utils tests for undefined commandVars
* Added some tests for the event configuration
* Revert event for HPOS e2e to push
* Use matrix name in artifact name to avoid duplication
* Test with pull_request event
* Use job-index for unique artifacts names
* Revert event for HPOS e2e to push
* Add api tests for HPOS disabled
* Use unique artifact name for api tests
* Revert event for HPOS disabled api tests to push
* Rebuild monorepo utils to fix a merge conflict
* Updated wording
* Fix plugin slug for reporting
* Comment out setting of woocommerce_coming_soon
* Add canvas utility
* Update create-cart-block.spec.js to use canvas instead of page
* Update create-checkout-block.spec.js to use canvas instead of page
* Extract common editor actions into editor utils
* Add changelog
* Update post and page creation tests to use canvas
* Update create-woocommerce-blocks.spec.js to use canvas
* Activate Gutenberg to test changes
* Remove plugins listing from env setup script
* Fix eslint errors
* Remove redundant env setup script call
* Install Gutenberg for test in CI
* Test with Gutenberg
* Remove Gutenberg
* Fix block editor variable product test
* Revert "Fix block editor variable product test"
This reverts commit cd004530a7.
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
* Add flaky test reporter for flaky e2e test reporter
* Add to stalebot workflow instead
* Update stale action to 9 for node 20
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Adrian Moldovan
Manish Menaria
Sam Seay
Jonathan Lane
jonathansadowski
Bart Kalisz
Moon
Luigi Teschio
nigeljamesstevenson
dependabot[bot]
Adrian Moldovan
Leif Singer