* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Add release checks jobs and deprecate smoke-test-release.yml
* Deprecate smoke-test-release.yml
* Add tests-release-checks.yml to trigger ci.yml on push to tags
* Add the pull_request trigger to test the workflow
* Remove the pull_request trigger to test the workflow
* Add the pull_request trigger to test the workflow
* Add the tag input in ci.yml
* Remove import and use core reference
* Only build if build-plugin is true
* Use release-artifact instead
* Split setup-monorepo step
* Use tag: 'nightly' to test
* Use inputs.tag directly in the wp-env setup step
* Add script to override the wp-env config
* Run the env config script in ci
* Set the artifact name
* Fix workflow syntax
* Add changelog
* Validate RELEASE_TAG and ARTIFACT_NAME
* Use real ref type and name
* Use real ref type and name
* Use real ref type and name
* Use github context instead
* Test with 9.0.0-beta.2
* Add push to tags trigger directly in ci.yml
* Configure one job to run with nightly-checks
* Remove empty lines
* Format events array one element per line
* Format events array one element per line
* Add flaky test reporting and issue creation for Blocks
This commit introduces the functionality for identifying and reporting flaky tests within our CI workflow.
- The reporter captures flaky test results and saves them to `flaky-tests` directory.
- If a test fails initially but passes upon retries, it is marked as flaky and a corresponding report is generated.
- The workflow creates GitHub issues for flaky tests, aiding in better visibility and tracking of these intermittent issues.
1. **Workflow Changes**:
- **Archive flaky test reports**: Adds a step to archive flaky test reports generated during the test runs.
- **Merge flaky test reports**: Introduces a new step to merge all flaky test reports from different shards.
- **Create GitHub issues for flaky tests**: Implements a job that uses the `@wordpress/report-flaky-tests` package to create GitHub issues for detected flaky tests when a PR is not in draft state and flaky test reports are present.
2. **New Flaky Test Reporter**:
- Adds `flaky-tests-reporter.ts` that defines a custom Playwright reporter for identifying and recording flaky tests.
3. **Playwright Configuration**:
- Updates `playwright.config.ts` to include the new flaky tests reporter in the list of reporters when running in CI.
4. **Demo Tests**:
- Adds `demo.spec.ts` containing sample tests to validate the flaky test reporting functionality.
Implementing this feature will help us track and address flaky tests more efficiently, ensuring more stable and reliable test suites.
* Remove draft condition for testing
* 'Merge failures artifacts' job shouldn't touch flaky test artifacts
* Try fixing if condition
* Add required permissions
* Let's try providing all the permissions
* Let's try providing all the permissions
* Create flaky tests issues when PR is merged with trunk
* Update if condition
* Run action for specific paths only
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Remove Demo file
* Remove Demo file
* Use commit hash to use exact version of the script
* Replace label with team: Kirigami & Origami
---------
Co-authored-by: github-actions <github-actions@github.com>
* Deprecate pr-smoke-test.yml
* Add configs for each environment
* Add scripts
* Add arguments to playwright command
* Update pnpm scripts and existing e2e projects
* Fix shellcheck warning
* Update test env start for Gutenberg stable run
* Add changelog
* Remove unused var
* Set the USE_WP_ENV var
* Configure daily e2e jobs
* Rename some steps
* Update PR template and auto change log
* Ignore .github md files in linting
* Rebuilt monorepo-utils to solve conflict
* Updated regex pattern to include uppercase X
* Move the changelog entry not required checkbox on top of the description.
* Return the comment even if shouldAutomateNoChangelog is false
* Updated tests
* Rebuilt monorepo-utils
---------
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
* Add the slack-test-report util
* Error if the required GitHub context variables are not set
* Error if the required SLACK_CHANNEL env var is not set
* Use a more generic message to be able to report on jobs that are not tests, like build
* Add the alert-on-failure job in ci
* Run for pull_request to test the job
* Setup monorepo first
* Force a linting error
* Better logging
* Use inputs.trigger as report name if set
* Force an e2e test failure
* Set the commit message in the notification
* Use INPUT_TRIGGER for all notifications
* Revert test changes
* Add condition on github.event_name for lint jobs
* Update a file to trigger a lint job
* Update condition to run only on pull_request
* Revert test change
* Always run the Slack notification step
* Add pull_request test trigger
* Update report step
* Remove branch filter
* Try with push event and branch filter
* Pass secrets to ci.yml
* Update testIgnore pattern for Gutenberg project
* Add changelog
* Remove the test push trigger
* One more test run
* Removed push test trigger
* Add workflow_call event
* Add tests-daily-run.yml
* Remove workflow version
* Update input value
* Remove secrets from caller workflow
* Rename job
* Check context
* Check context
* Check workflow_call inputs
* Override github.event_name with inputs.trigger
* Fix input name
* Add config for daily e2e tests with Gutenberg
* Update env commands for Gutenberg run
* Add disableHpos env variable in ci-utils
* Revert the trigger to push
* Revert the env variable setting in ci-jobs
* Revert the env variable setting in ci-jobs
* Rebuild utils
* Update test env start command
* Define the Gutenberg Playwright project
* Use path relative to workspace root
* Update events for test projects
* Update events for test projects
* Add changelogs
* Remove the e2e tests with Gutenberg tests from daily runs
* Set the schedule and workflow_dispatch triggers
* Add --project default flag to only run the default tests in CI
* Renamed snapshots to match the new Playwright project name
* Add disableHpos test env variable
* Update events for HPOS disabled tests
* Add pull_request event to test the env
* Remove the pull_request event
* Add changelog
* Make the disableHpos var a boolean
* Also set the env in ci.yml
* Remove console log statement
* Print woocommerce_custom_orders_table_enabled
* Run test setup script again to check output
* Capture env setup output and print it
* Get woocommerce_custom_orders_table_enabled value in test env script
* Get more details on HPOS setting
* Remove the DISABLE_HPOS logic from test env setup script
* Use DISABLE_HPOS instead of ENABLE_HPOS everywhere
* Turn off no-console and jest/no-standalone-expect eslint rules for api core tests
* Test API tests with HPOS disabled
* Fix eslint error
* Clean up debug statements
* Remove pull_request test trigger
* Include performance tests job in the needed jobs for evaluation
* Use Github api to get the workflow jobs status
* Evaluate project-jobs
* See results
* Check for cancelled jobs
* Add evaluate-project-jobs.js and rename optional jobs
* Fix script path
* Checkout first
* And then fix the path again.
* Update env variable name and be more detailed in what are the required variables
* Use a flat matrix env variable
* Update to trigger test jobs
* Use the jobs name to evaluate optional requirement
* Fix conditions
* Prettier print
* Test lint job failure
* Remove unused MATRIX env variable
* Fix test
* Remove unused MATRIX variable check
* Only run my account e2e tests
* Remove unused function
* Nicer console printing
* Revert change that triggers failing lint job
* Force failing e2e test
* Revert e2e test command and forced failure
* Added test data and test mode
* Added more test data
* Fixed lint errors and warnings
* Exclude .github folder from eslintignore
* Change to trigger everything
* Revert change to trigger everything
* try db reset in page teardown
* move reset to setup step
* Use wp db cli
* Fix global setup
That part is overriding the logged in user state and wipes out the nonce and rootUrl fields.
* Try importing instead from a generated dump
* Revert "Try importing instead from a generated dump"
This reverts commit 987dc471c9.
* Revert "Revert "Try importing instead from a generated dump""
This reverts commit c8d008cb20.
* Don't bypass visitSiteEditor so that the Welcome Guide is closed
* use createNewPost
* Revert "Revert "Revert "Try importing instead from a generated dump"""
This reverts commit 2684273582.
* [Blocks]: Fix E2E tests (#46344)
* Load local pickup enabled setting as bool not string
* Add changelog
* Remove unused import
* Update plugins/woocommerce-blocks/assets/js/extensions/shipping-methods/pickup-location/utils.ts
Co-authored-by: Niels Lange <info@nielslange.de>
* Use strict equality test
* try now
* try now
* add a new sql file
* remove default.sql
* fix welcome guide tour
* fix E2E tests
* Fix the template revert tests
...where the template is unreachable due to pagination.
* Add changelog entry
* improve logic to closeWelcomeGuideModal
* fix cart checkout tests
* improve flakiness
* improve flakiness
* fix flaky test
* fix company field
* fix E2E test
* fix E2E tests
---------
Co-authored-by: Thomas Roberts <thomas.roberts@automattic.com>
Co-authored-by: Thomas Roberts <5656702+opr@users.noreply.github.com>
Co-authored-by: Niels Lange <info@nielslange.de>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
* Replace beforeAll w/ beforeEach + remove all after* hooks
* Fix broken hooks
* Activate plugins via requestUtils API
* [Blocks - E2E]: Add `playwright/no-hooks` ESlint rule (#46432)
add ESLint configuration
* Clean up console logs
* Remove obsolete language setup steps
* Remove more unnecessary setup steps
* Remove even more obsolete setup steps
* tmp: add the LYS fix
* Try stabilizing the company field test
* Use canvas param instead of manually entering edit mode
* Remove double site editor redirect
* Revert "Use canvas param instead of manually entering edit mode"
This reverts commit 5e6cc17154.
* Revert "Remove double site editor redirect"
This reverts commit 69a57a82a8.
* Fix flaky products sorting test
* Blocks: Fix ESLint errors (#46595)
fix eslint error
* Fix ESLint errors (#46626)
* Fix ESLint errors
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
---------
Co-authored-by: github-actions <github-actions@github.com>
* Blocks: Migrate multiple tests to Playwright
* fix path
* migrate other tests
* add product best sellers E2E tests
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* migrate jest-puppetter E2E tests to Playwright
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Blocks: Delete Jest-Puppeteer infrastructure
* Add changefile(s) from automation for the following project(s): woocommerce-blocks, woocommerce
* Remove obsolete waitForLoadState call
* improve E2E tests
* fix E2E tests
* improve E2E tests
* remove old test
* remove not necessary test
* fix number product
* fix E2E test
* fix number of products
* fix product count
* refactor store notice E2E tests
* update string
* fix E2E test
---------
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Thomas Roberts <thomas.roberts@automattic.com>
Co-authored-by: Thomas Roberts <5656702+opr@users.noreply.github.com>
Co-authored-by: Niels Lange <info@nielslange.de>
Co-authored-by: github-actions <github-actions@github.com>
* Update env setup script to disable hpos
* Use DISABLE_HPOS instead of ENABLE_HPOS
* Log HPOS state
* Configure project for core e2e with HPOS disabled
* Add event argument for ci-jobs util
* Add event option for test jobs
* Add support for github events
* Add changelog
* Configure HPOS tests to run on pull_request
* Fix utils tests for undefined commandVars
* Added some tests for the event configuration
* Revert event for HPOS e2e to push
* Use matrix name in artifact name to avoid duplication
* Test with pull_request event
* Use job-index for unique artifacts names
* Revert event for HPOS e2e to push
* Add api tests for HPOS disabled
* Use unique artifact name for api tests
* Revert event for HPOS disabled api tests to push
* Rebuild monorepo utils to fix a merge conflict
* Updated wording
* Fix plugin slug for reporting
* Comment out setting of woocommerce_coming_soon
* Add canvas utility
* Update create-cart-block.spec.js to use canvas instead of page
* Update create-checkout-block.spec.js to use canvas instead of page
* Extract common editor actions into editor utils
* Add changelog
* Update post and page creation tests to use canvas
* Update create-woocommerce-blocks.spec.js to use canvas
* Activate Gutenberg to test changes
* Remove plugins listing from env setup script
* Fix eslint errors
* Remove redundant env setup script call
* Install Gutenberg for test in CI
* Test with Gutenberg
* Remove Gutenberg
* Fix block editor variable product test
* Revert "Fix block editor variable product test"
This reverts commit cd004530a7.
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
* Add flaky test reporter for flaky e2e test reporter
* Add to stalebot workflow instead
* Update stale action to 9 for node 20
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
* Updated deprecated gh-script action to node 20 variant
* Minor change in e2e test to trigger CI
* Revert e2e test change
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
* Move daily site check workflow to deprecated
* Add install-plugin script
* Install plugins script
* Fix the script path, it was moved.
* Set env on job level
* Update working-directory
* Add changelog
* Use the right WC_SUBSCRIPTIONS_REPO variable
* Add a test channel id
* Update smoke-test-daily.yml
* Remove WC Subscriptions
* Update smoke-test-daily.yml
* Switch back the channel ID and remove pull_request trigger
* Move deprecated workflows to deprecated folder
* Configure new test job for metrics
* Add changelog
* Update start command
* Update start command
* Archive metrics results
* Fix script path
* Fix name in archive results condition
* Add the actual run command for pull_request event
* Run with pnpm --filter
* Test run for push event
* Revert test conditions
* Add WP_ARTIFACTS_PATH env variable
* Update path to readme.txt
* Revert github event conditions
* Add CODEVITALS_PROJECT_TOKEN env
* Set WP_ARTIFACTS_PATH on job level
* Add blocks templates in the changes list
* Update code freeze GH Actions workflow to upload the woocommerce packages to the R2 bucket on code freeze.
* Make the job that uploads WooCommerce release packages to the R2 bucket independent from the Slack integration in release-code-freeze.yml workflow.
* Revert "[e2e tests blocks] Configure the number of shards for each matrix project (#45212)"
This reverts commit 72789656cf.
* trigger CI
* run E2E tests
* restore ci.yml file
* trigger E2E tests
* disable E2E tests about mini cart translations
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Fix the setupAndFetchQueryContextURL utility
* add comment
* Remove obsolete test
* Blocks E2E: Update timeout values (#45353)
Set timeout values same as core
---------
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
* Update workflow to use latest checkout actions
* Update non-HPOS workflow as well
* Update additional actions
* Set a slow test threshold at 30 seconds (15 is default)
* Final action updates
* Slow tests has to be an object
* Pin external actions to commit
* Slow test threshold and max
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
* exclude dependabot from commmunity contributor label
* use prior art from block repo
---------
Co-authored-by: Ron Rennick <ronald.rennick@automattic.com>
* Add sharding to Blocks e2e tests
* Changelog
* Push a change to Blocks README to trigger tests
* Fix syntax
* Fixed again
* Try again
* Trying to get the shard number
* Shard names
* fix E2E test
* complete all the jobs
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Luigi Teschio <gigitux@gmail.com>
* quote lines starting with asterisk
* use single quotes instead of double quotes
* try triggering the workflow locally
* remove `workflow_dispatch` trigger again -- temporary testing
* Skip workflows when only .md files are updated
* Add paths-ignore for performance tests
* Add paths-ignore in the pr merge workflow
* Include docs/** in the ignore paths
* Update the list of paths to ignore
* Update the list of paths to ignore
* Remove docs plugin from monorepo
* Add language to md-docs readme
* Fix lint error in blocks
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
---------
Co-authored-by: github-actions <github-actions@github.com>
* Update commands to run e2e tests
* Capitalize Product Catalog template name
* Create tests with block theme with templates
* Create Block Theme with Templates test environment
* Update classic theme commands
* Code cleanup
* Add changefile(s) from automation for the following project(s): woocommerce-blocks
* Code cleanup (II)
* Add config to blocks-playwright.yml and remove unnecesssary .github/workflows/playwright.yml file
* Remove unnecessary imports from product-catalog-template.block_theme_with_templates.spec.ts
* Update blocks e2e commands to pnpm
* Replace hardcoded strings with a variable
* Unify all 'block theme with templates' tests in a parameterized test
* Create a testData object for the Mini-Cart template part test
* Fix some code styling issues
* Create a testData object for the Mini-Cart template part test (II)
* Remove duplicate userText variable
---------
Co-authored-by: github-actions <github-actions@github.com>
* Change HPOS and non HPOS e2e tests to use 5 shards each
* Add changelog
* Update numbering
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Unfortunately, it looks like we need to build in order
to be able to lint. This makes sense since some
projects will lint imports from others that haven't
been built yet.
This adds support for using the `pnpm utils ci-jobs` command in our `ci.yml` file. One of the bigger benefits to this change too is that we're now distributing a bundled version of the utils tool. This lets us run it without actually having to install the repo and will let us speed up any workflows that currently do.
* Try sharding e2e tests
* Update reporter for merge
* Add changelog
* Change to trigger CI
* Update job name to match existing run
* Install with dependencies
* Set working directory
* Only run 2 workers at a time on CI
* Try single run, but 6 shards
* Simplify welcome modal closing util helper
* Switch to more simpler way of closing welcome modal
* Fix flakiness in create shipping zone test
* Try 10 shards
* Drop CI retries to 2 from 4
* Update path to reports
* Fix syntax error in GH file
* Remove working directory
* Update directories
* Path again
* Use Allure dir
* Fix merge step
* Install dependencies
* Setup repo in report merge
* Commands to merge reports
* Set working directory
* Update actions
* Name upload artifacts
* Just numbers for shards
* Update matrix
* Need to redeclare env variables
* Update file paths for merge step
* Remove working directory for merge step
* Try absolute paths
* Missed a path
* Put all artifacts in single download folder
* Tweak download action
* Might not have to merge reports
* Fix typo
* Update path for summary report
* Update results paths
* Tweak condition
* Add step to check success of matrix run
* Remove ternary per feedback
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Veljko <veljano@yahoo.com>
* Bump pnpm version in the release-code-freeze.yml workflow to fix issue that impeded the creation of new WooCommerce zip packages.
* Update code freeze workflow to automatically get the pnpm version from the root package.json file.
* Improve reusability of the method to retrieve the pnpm version in the Code Freeze workflow.
* First pass at Playwright test migration
* Set working directory
* Remove exec
* Configure the environment before testing
* Remove filter
* Update paths for tests
* Try explicitly stating working-directory
* Change tests directory
* Try running all tests
* Uncommented project
* Update commands to run tests
* Comment out env config
* Uncomment
* Changelog
* Set up multiple test configurations
* Update failing test
* Update paths for artifacts
* Update path to artifacts
* Path to results
* remove unnecessary file
* Skip translation E2E tests
Skipping translation tests until we fix the missing translations issues after we changed the domain to "woocommerce"
* Update the artifacts path
* Fix lint errors
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Luigi Teschio <gigitux@gmail.com>
Co-authored-by: Saad Tarhi <saad.trh@gmail.com>
* Migrate Puppeteer test action from Blocks repo
* Update file path
* Use pnpm
* Use npm
* Updates based on feedback
* Comment out paths for testing
* Fix paths with working directory
* Use pnpm
* Fix missing pnpms
* npm instead
* Do use pnpm
* Update command
* Blocks wp-env config
* Update paths, config files
* Run test commands from blocks
* Config update
* Uncomment paths so that e2e checks only run on Blocks
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
* Update action for API tests
* Changelog
* Update e2e tests
* Rename WooCommerce Payments to WooPayments
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Adrian Moldovan
Manish Menaria
Sam Seay
Jonathan Lane
jonathansadowski
Bart Kalisz
Moon
Luigi Teschio
Ron Rennick
nigeljamesstevenson
dependabot[bot]
Adrian Moldovan
Leif Singer
Job
Alex Florisca
Alejandro López Ariza
Paul Sealock
Albert Juhé Lluveras
Christopher Allford
Riad Benguella