Commit Graph

110 Commits

Author SHA1 Message Date
nigeljamesstevenson 3568e6f7a9
Revert "k6 perf tests: Fix parsing of COT/HPOS environment variable" 2023-11-07 11:29:35 +00:00
Rodel Calasagsag 22a1aabdd4 Remove enabling HPOS via old external plugin 2023-10-22 21:11:41 +08:00
nigeljamesstevenson 026c3ea20d
Ensure HPOS is disabled if ENABLE_HPOS is set to '0' (#40528)
* Ensure HPOS is disabled if ENABLE_HPOS is set to '0'

* update spacing and changelog

* log ENABLE_HPOS value for GitHub action

* remove log ENABLE_HPOS value for GitHub action

* updates to HPOS using REST API

* update spacing

* code tidy up

* workflow and global-setup updates after review

* update workflow for api tests

* workflow updates

* delete .github/workflows/pr-build-and-e2e-tests-non-hpos.yml

* update workflow

* update api setup for HPOS

* updates to workflows for api references and setup updates

* add in ENABLE_HPOS check

* update API tests BASE_URL

* updates after review

* update error text

* remove extraneous variable names
2023-10-11 15:03:24 +08:00
Paul Sealock 7164093928
Workflows: Remove Github Actions Bot conditionals (#38438) 2023-05-25 12:00:22 +12:00
Sam Seay 46aa3f2aa1
Add check for bot author to steps that should be skipped (#37949)
Add check for bot author to steps that should be skipped.
2023-05-02 15:21:35 +12:00
Alejandro López Ariza c3994e6f9d
Add workflow to remind PR reviewers to also review the testing instructions. (#36918)
* Add workflow to remind PR reviewers to also review the testing instructions.

* Update review-testing-instructions GitHub Actions workflow to also mentioning teams in the comment.

* Update review-testing-instructions GitHub Actions workflow to find if the comment it posts was already posted and - in that case - update it instead of creating a new one.

* Pin find-comment and create-or-update-comment thirdparty GitHub Actions to full length commit SHA instead of tag for better security.
2023-03-22 12:50:36 +01:00
jonathansadowski 4ebd35b41d
Update automation bypass to work correctly, consider actions bot. (#37045) 2023-03-09 17:10:33 -06:00
Sam Seay 262b2ecebd
Don't run CI checks on PRs that are just changelog changes (#36280) 2023-01-12 14:09:36 +13:00
Roy Ho 16386238cc
Add minimum required permissions for workflow token (#36200)
* Add minimum required permissions to e2e-tests

* Add minimum token permission to more workflows
2023-01-02 09:28:11 -08:00
rodelgc c4260c935f
Consistent folder structure for E2E and API test results (#35907)
Consistent folder structure for E2E and REST API test results
2022-12-19 14:33:22 +08:00
Sam Seay 43143b848f
Remove some usages of set-output (#35799)
* Migrate straight script usage of set-output to write to GITHUB_OUTPUT
* Use file_put_contents in php scripts to set GITHUB_OUTPUT
2022-12-06 09:35:49 +13:00
Jonathan Lane fd7fc355fa
Fix/e2e environment test failures (#35679) 2022-11-22 16:16:42 -07:00
Alejandro López Ariza 4cb9bc45c4
Updated COT plugin used to set up the local environment with COT enabled. (#34990)
* Updated COT plugin used to set up the local environment with COT enabled.

* Enable HPOS in the e2e environment using env var (#35057)

* Updated COT plugin used to set up the local environment with COT enabled.

* Used env var to enable HPOS

* Disable HPOS for performance tests

* Updated permissions

* Set ENABLE_HPOS to 1

Co-authored-by: Alex López <alex.lopez@automattic.com>

Co-authored-by: Jamel Noel Reid <MrJnrman@users.noreply.github.com>
2022-10-26 09:29:41 -07:00
rodelgc 39b472be7a
Set paths for `allure-results`, `test-results.json`, and save state files to be inside their respective E2E or API folders (#35206)
* Set paths to e2e-pw

* Delete and untrack storage state files

* Add changelog

* Checkout updated version of smoke test daily workflow

* Allow setting allure output paths to e2e-pw folder using environment variables

* Set allure output paths to be inside api-core-tests/api-test-report folder

* Remove unnecessary TODO comment
2022-10-24 16:53:46 -07:00
Jonathan Lane aa438e5741
Fix/update playwright and test fixes (#34790)
* Update Playwright 1.22.1 -> 1.26.0

* Update page title locator

* Wait for View cart link after adding to cart

* Add a wait when applying second coupon

* Fix flakey coupon test

* Ensure Store is in US

* Add changelog

* Revert lock file commit

This reverts commit b75192f53c.

* Updated lock file

* Added command for Playwright test execution

* Resolve merge conflict

* Resolve merge conflict

* Fixed pnpm-lock.json

* Update PW Selectors (#34959)

* Updated PW selectors

* Updated selector

* Removed extra command

* Ensure Billing country is set

* Fix failing payment setup test

* Old test is still running in actions

* Adds colour output for Playwright in GH Actions

* Fix payment test

* Lock file update

* Skipping a test for now

Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: jamelreid <jnoelreid@gmail.com>
Co-authored-by: Jamel Noel Reid <MrJnrman@users.noreply.github.com>
2022-10-07 12:39:39 -06:00
rodelgc f8ea46b086
Implement `maxFailures` and warning messages (#33977)
* Add `maxFailures`

* Intentionally fail a test for testing the failure limit

* Add changelog

* Add step to get total number of E2E tests

* Increase max failures

* Add warnings to the test summary.
Also moved some stuff around.

* Update changelog

* Use env var to set `maxFailures` value instead of hard-coding it

* Make sure env var is converted to a number

* Allow maxFailures even if not on CI

* Show incomplete e2e run warning only when E2E_MAX_FAILURES is set

* Safely escape new lines in comment body

* Fix returned stringified summary

* Remove unnecessary newline escaping

* Fix incorrect reference to E2E_MAX_FAILURES

* Minor formatting change

* Shorten warning messages

* Minor change in warning message

* Intentionally fail only 1 test

* Remove intentional failures

* Removed duplicate env declaration

* Correct pnpm command and playwright config file location

* Simplify reference to CI variable

* Remove unnecessary if condition

* Remove references to the old E2E_PLAYWRIGHT flag

* Remove references to the old E2E_PPTR_SUMMARY_PATH variable

* Test for 2 warnings

* Revert to correct locator

* Test for only 1 warning

* Pass all tests
2022-10-05 10:44:20 -07:00
nigeljamesstevenson 456dd02352
update workflows to use the playwright api-core-tests instead of the … (#34935)
* update workflows to use the playwright api-core-tests instead of the equivalent supertest versions

* add changelog file
2022-10-03 10:49:42 -06:00
nigeljamesstevenson 75c91ccdc0
E2e pw/migrate api core tests to use playwright 34126 (#34835)
* playwright api-core-tests

* playwright api-core-tests

* playwright api-core-tests

* Workflow updates

* Workflow test updates

* workflow test update

* Workflow testing

* Workflow testing

* workflow test updates

* workflow test updates

* Workflow update test

* Workflow update test restore

* add gitignore for api core tests output dir

* Workflow test with playwright api tests

* Workflow restore to original

* update readme and playwright config
2022-10-03 07:41:27 -06:00
Sam Seay 04b74c1053
Upgrade to pnpm 7 (#34661)
* Remove -- -- from scripts that fixed issue in pnpm 6, fix a couple TS issues in sep packages
* Minor fixes to analyzer scripts and doco based on pnpm 7
* Add dompurify types to data package to avoid TS errors
* Remove pnpx in favor of pnpm exec
* Modify the code analyzer to respect pnpm version if present.
* Update instructions for running recursive lint, add comments to explain
2022-09-16 09:21:42 +12:00
Christopher Allford a49f23abfd
Optimize Workflow Caching (#34607)
* Added Monorepo Setup Action

This action will handle the installation, building, and caching for all
projects within the monorepo. It has inputs for skipping builds and
filtering so that only specific packages are installed and built.

* Removed Test Caching

Caching the results of tests, while an interesting way to avoid
unnecessary execution, seems error-prone. We can't adequately
capture the environment such that we can rely on this working.
For instance, changes in PHP version might break PHPUnit tests,
but, cached runs will be shared across different workflows.

It seems better to just not cache it and rely on people only testing
code that has actually changed in the pull request.

* Added Input Parsing

This SHOULD allow for passing multiple filters as an array argument.

* Updated Workflows

This fully utilizes the new action across all of the workflows.

* Always Cache Build Output

Since there may be other commands run that populate the cache, we
should make sure that we are always caching just in case.
2022-09-13 13:55:03 +12:00
Jamel Noel Reid 40dbc1f924
Update CI to use wp-env for api, e2e and performance tests (#34311)
* Renamed e2e to e2e-legacy

* Moved e2e folder to tests/e2e

* Updated gitignore

* Updted init script to work with wp-env

* Updated readme

* Updated commands to match new test structure

* Updated workflow to work with new test structure

* Added changelog

* turned off alternate wp cron in test

* Fixed path to playwright config

* Renamed e2e to e2e-pw

* Renamed e2e-legacy to e2e

* Updated e2e-pw path

* Fixed workflow

* Fixed csv test

* Updated .gitignore

* Removed puppeteer specific step
2022-08-15 11:47:53 -07:00
Lucas Bustamante edd453c40b
Skip test result generation and upload in forks (#34023)
* Skip test result generation and upload in forks

* Generate test report in fork
2022-07-21 11:49:25 -06:00
Lucas Bustamante 4ffd5992a9
Reduce e2e flakiness in CI (#33979) 2022-07-19 11:28:15 -06:00
rodelgc 9e576c2208
Upload API and E2E test reports to S3 bucket (#33843)
* Simplify if expression

* Call the new workflow in Woo Test Reports repo for publishing PR test reports

* Rename Playwright-related steps

* Archive Puppeteer screenshots only when Playwright flag is off

* Correct step name and if condition in Puppeteer steps

* Run only a single Playwright test for testing

* Correct workflow name

* Add commit message as input

* Remove invalid COMMIT_MESSAGE input

* Add root_dir input

* Update s3 root dir input

* Update link to full E2E test report

* Re-enable all Playwright tests
2022-07-14 14:15:20 -07:00
Jonathan Lane 0cebdf5277
Convert e2e tests to Playwright (#33462)
* Final updates for Playwright

* Update config

* Add uuid dependency

* Increase retries to 2

* Update selectors on shipping page

* Use baseURL instead of hard-coded URL for API

* Clarify comment

* Use baseURL instead of hard-coded URL

* Check to see if an order was created before attempting to delete it

* Add changelog

* Turn on Playwright tests in GitHub

* Increase timeout for CI execution

* Update configuration options (minor edit)

* Fix for checkout flaky test

* Parse orderId from URL

* Check for substring

* Streamline email tests

* Remove .only

* Only clear email logs of messages for test

* Get orderId from page element

* Fix for test not waiting for reset

* Add in second wait for Performance section

* Change significance from minor to patch

Co-authored-by: Jon Lane <jon.lane@automattic.com>
2022-07-06 14:28:00 -06:00
rodelgc aed4763414
Post test result summary in workflow run and in PR comment using github-actions[bot] (#33675)
* Temporarily toggle Playwright feature flag on

* Add job for test summary

* Add dummy run command

* Debug object contents

* Test getting json from prev step output

* Try toJSON()

* Test access to keys

* Try using context

* Use separate file

* Debug steps

* Try dummy job summary

* Minor job rename

* Add script for preparing job summary

* Fix context syntax error

* Correct artifact path

* List contents of artifacts folder

* inspect API summary contents

* inspect apisummary

* Remove dummy step for listing artifacts folder contents

* Parse number to string

* Correct SHA

* Try printing job summary on next step

* Create PR comment

* Minor step rename to test PR comment update

* Add another line break

* Test for puppeteer tests

* Test full Playwright test run

* Test full Puppeteer e2e run

* Fix spacing inconsistency

* Specify other test result categories in PPTR run

* Rename script file

* Minor refactor and documentation on script file

* Minor addition to script documentation
2022-07-06 11:17:51 -06:00
Jon Lane cb36fe8d50 Change flag back 2022-05-25 10:21:17 -07:00
Jon Lane 06d0e7a876 Test feature flag for Playwright 2022-05-25 10:19:50 -07:00
Rodel f69d8ba6cb Toggle flag off 2022-05-24 14:45:02 +08:00
Rodel 54ff88ffd7 Change step name 2022-05-24 14:44:10 +08:00
Rodel ed70a94a07 Install chromium browser only 2022-05-24 14:06:23 +08:00
Rodel a84d96afc7 Remove extra spaces 2022-05-24 13:31:59 +08:00
Rodel cb1910ab18 Change flag scope to workflow level 2022-05-24 13:27:53 +08:00
Rodel d1a5df46e6 Correct trigger for "Archive e2e report" step 2022-05-24 13:13:48 +08:00
Rodel fb51e5dfc2 Use flag on publish reports job 2022-05-24 13:11:12 +08:00
Rodel f279202ee8 Improve readability of if condition 2022-05-24 12:47:00 +08:00
Rodel 89a3924512 Remove unnecessary step id 2022-05-24 12:37:54 +08:00
Rodel abd4c5e683 Correct if conditions 2022-05-24 12:36:57 +08:00
Rodel c9ebaa22c2 Set flag to literal boolean value 2022-05-24 11:19:52 +08:00
Rodel 2a6554be94 Change variable assignment 2022-05-20 16:58:29 +08:00
Rodel 3407b2d0ad Remove unnecessary brackets 2022-05-20 16:39:38 +08:00
Rodel 08d34968ce Try correcting boolean value assignment 2022-05-20 16:38:59 +08:00
Rodel 40d04ff1b8 Ignore archiving of playwright test report when files were not generated 2022-05-20 16:32:04 +08:00
Rodel 4474e08ff9 Correct bad line break 2022-05-20 16:19:57 +08:00
Rodel 018677e35b Add step to generate e2e report on CI 2022-05-20 16:08:49 +08:00
Rodel b2a61a760c Add step to install playwright supported browsers 2022-05-20 16:05:12 +08:00
Rodel 0c826bf22d Change VM in Publish test reports job 2022-05-20 14:41:49 +08:00
Rodel a53e61c836 Implement Playwright feature flag in PR test workflow 2022-05-20 14:37:25 +08:00
rodelgc 7e49c151b7
Update publish-test-reports if condition (#33085)
Prevent running the "Publish test report" job when none of the tests ran
2022-05-19 11:20:33 +08:00
roykho 1042f22cf8
Extract cache and build to rest of workflows 2022-05-17 11:48:24 -07:00