In this PR, we are re-iterating the job speedup and leveraging single wp-instance, incremental builds, and baseline reports caching. The original benchmarking tool is still available but bypassed as we evaluate results for this iteration approach.
* Include performance tests job in the needed jobs for evaluation
* Use Github api to get the workflow jobs status
* Evaluate project-jobs
* See results
* Check for cancelled jobs
* Add evaluate-project-jobs.js and rename optional jobs
* Fix script path
* Checkout first
* And then fix the path again.
* Update env variable name and be more detailed in what are the required variables
* Use a flat matrix env variable
* Update to trigger test jobs
* Use the jobs name to evaluate optional requirement
* Fix conditions
* Prettier print
* Test lint job failure
* Remove unused MATRIX env variable
* Fix test
* Remove unused MATRIX variable check
* Only run my account e2e tests
* Remove unused function
* Nicer console printing
* Revert change that triggers failing lint job
* Force failing e2e test
* Revert e2e test command and forced failure
* Added test data and test mode
* Added more test data
* Fixed lint errors and warnings
* Exclude .github folder from eslintignore
* Change to trigger everything
* Revert change to trigger everything
* Move deprecated workflows to deprecated folder
* Configure new test job for metrics
* Add changelog
* Update start command
* Update start command
* Archive metrics results
* Fix script path
* Fix name in archive results condition
* Add the actual run command for pull_request event
* Run with pnpm --filter
* Test run for push event
* Revert test conditions
* Add WP_ARTIFACTS_PATH env variable
* Update path to readme.txt
* Revert github event conditions
* Add CODEVITALS_PROJECT_TOKEN env
* Set WP_ARTIFACTS_PATH on job level
* Add blocks templates in the changes list
* exclude dependabot from commmunity contributor label
* use prior art from block repo
---------
Co-authored-by: Ron Rennick <ronald.rennick@automattic.com>
This adds support for using the `pnpm utils ci-jobs` command in our `ci.yml` file. One of the bigger benefits to this change too is that we're now distributing a bundled version of the utils tool. This lets us run it without actually having to install the repo and will let us speed up any workflows that currently do.
* use latest commit of stalebot action, tweak configuration accordingly
* remove stalebot.js
* use default operations per run
---------
Co-authored-by: Ron Rennick <ronald.rennick@automattic.com>
* change reference of Catpure to Capture
Co-Authored-By: Vikram <93216400+vikrampm1@users.noreply.github.com>
* change reference of expicitly to explicitly
Co-Authored-By: Vikram <93216400+vikrampm1@users.noreply.github.com>
* change reference 'cutted' to 'cut'
* change reference 'determening' to 'determining'
* change reference 'retreive' to 'retrieve'
* change reference 'neccessary' to 'necessary'
* change reference 'Fitler' to 'Filter'
* change reference of "seperate" to "separate"
Co-Authored-By: Ankit K Gupta <ankit.himcs@gmail.com>
* change reference of "wether" to "whether"
Co-Authored-By: Sumit Bagthariya <67687255+qasumitbagthariya@users.noreply.github.com>
* change reference of "staus" to "status"
* change reference of "retrive" to "retrieve"
* change references of "gatways" to "gateways"
* change references of "existant" to "existent"
* change reference of "requries" to "requires"
* change reference of "configuation" to "configuration"
* change reference of "processsing" to "processing"
* change reference of "represenation" to "representation"
* change reference of "dimentions" to "dimensions"
* change references of "reigster" to "register"
* change reference of "colum" to "column"
* change reference of "transtions" to "transitions"
* change references of "intially" to "initially"
* change references of "orignal" to "original"
* change references of "deprected" to "deprecated"
* change references of "paramter" to "parameter"
* change reference of "intance" to "instance"
* change reference of "elemets" to "elements"
* change references of "funcitons" to "functions"
* change reference of "specificed" to "specified"
* change references of "atributes" to "attributes"
* change reference of "tast" to "task"
* change reference of "chaning" to "changing"
* change reference of "retreiving" to "retrieving"
* change reference of "caluclation" to "calculation"
* change references of "Invaid" to "Invalid"
* change references of "paramaters" to "parameters"
* change reference of "Additonal" to "Additional"
* change reference of "teh" to "the"
* change reference of "evalutes" to "evaluates"
* change reference of "addedd" to "added"
* change reference of "excempt" to "exempt"
* change reference of "sequencially" to "sequentially"
* change reference of "previos" to "previous"
* change reference of "elegible" to "eligible"
* change references of "Boostrap" to "Bootstrap"
* change references of "compability" to "compatibility"
* change reference of "heirarchy" to "hierarchy"
* change references of "visibilty" to "visibility"
* change reference of "comparsion" to "comparison"
* change reference of "capabilties" to "capabilities"
* change reference of "datatores" to "datastores"
* change reference of "occured" to "occurred"
* change reference of "coresponding" to "corresponding"
* change references of "thier" to "their"
* change reference of "sucessfully" to "successfully"
* change reference of "insde" to "inside"
* change reference of "nagivation" to "navigation"
* change references of "visiblity" to "visibility"
* change reference of "documentaiton" to "documentation"
* change reference of "anayltics" to "analytics"
* change reference of "intalling" to "installing"
* change reference of "mininum" to "minimum"
* change references of "intial" to "initial"
* change reference of "Feld" to "Field"
* change reference of "taks" to "task"
* change reference of "trasnfer" to "transfer"
* change reference of "respone" to "response"
* change reference of "Extenstions" to "Extensions"
* change reference of "detault" to "default"
* change reference of "simultanious" to "simultaneous"
* change reference of "overides" to "overrides"
* change references of "Indvidual" to "Individual"
* change reference of "refering" to "referring"
* change reference of "aginst" to "against"
* change reference of "execuatable" to "executable"
* change reference of "repsonse" to "response"
* change reference of "documention" to "documentation"
* change reference of "asumed" to "assumed"
* change reference of "Minium" to "Minimum"
* change reference of "unqiue" to "unique"
* change references of "existance" to "existence"
* change reference of "compatability" to "compatibility"
* change reference of "Taxnomy" to "Taxonomy"
* change reference of "quering" to "querying"
* change reference of "retrun" to "return"
* change reference of "informations" to "information"
Co-Authored-By: Viktor Szépe <viktor@szepe.net>
* Add changelog
* Add changelog
* Fix typo
---------
Co-authored-by: Vikram <93216400+vikrampm1@users.noreply.github.com>
Co-authored-by: Ankit K Gupta <ankit.himcs@gmail.com>
Co-authored-by: Sumit Bagthariya <67687255+qasumitbagthariya@users.noreply.github.com>
Co-authored-by: Viktor Szépe <viktor@szepe.net>
Co-authored-by: Chi-Hsuan Huang <chihsuan.tw@gmail.com>
* First pass at updating release test workflow
* Add changelog
* Set dir env variables
* Update to workflow
* Fix indent
* Fix indent
* Clean up indent
* Re-order steps
* Change order of jobs
* Added common php versions
* Update pipeline
* Update some labels
* Simplify for testing
* Update paths
* Create tmp folder
* Fix path
* Paths
* Try outputting some debugging
* Add step ID back
* Remove working directory
* Another path tweak
* Add API release tests
* Add k6 tests
* Add PHP tests
* Launch wp-env during PHP tests
* Try default values
* Tweak some settings, add WP testing
* Tweak some settings
* Re-order e2e steps
* Update step descriptions
* Reorganize tests, add plugin tests
* Enable only e2e job
* Initial set up to run against release smoke test site
* Fix syntax
* Temporarily disable update wc spec
* Temporarily disable downloading woocommerce zip
* Download release zip using tag name
* Fix wrong job name
* Fix wrong job name
* Fix dir
* Delete fetch-asset-id.js
* REfactor update-woocommerce spec
* Add error handling
* Download release zip by tag
* Refactor update woo spec to download zip by tag
* Correct job name
* fail test on invalid tag
* Enable all e2e tests
* Run api tests before e2e tests
* Fix job dependency
* Add customer credentials to api job
* Separate job for WC Update
* Combine e2e allure-results, then report
* Enable report job
* Fix context
* Change job and artifact names
* Use test s3 path
* Minor job name change
* Upload artifacts to bucket
* Correct s3 path
* Add quiet option
* Retain video on failures
* Finalize s3 path
* Try WP latest-1
* Revert to wp latest
* Refine search for woocommerce zip asset
* Get created-at
* Specify repo in gh command
* Slugify env description
* Trim space
* Sync with bucket instead of copy
* Remove invalid --recursive flag
* Re-add missing step to combine e2e results from update wc test
* Ensure artifact upload on test failure
* Enable all e2e tests on WP latest
* Retain existing data before updating WC
* Make test compatible with 'Canceled' and 'Cancelled'
* Set env_desc as env var
* Re-add deleted file
* Fix UPDATE_WC in daily smoke test workflow
* Add tracing in global setup
* Remove tracing
* Temporarily run only basic spec
* Job for WP Latest-1 & 2
* Fix "Required input 'created_at' not provided"
* Minor rename
* Remove install filter
* Install deps in get-wp-matrix
* Delete get-wp-versions.js
* Add get-wp-versions.js to e2e-pw folder
* REname file
* REfactor
* Refactor script for getting WP prev versions
* Update job dependencies
* Temporarily remove disabled jobs
* Allow e2e-wp-latest after api test failure
* Update L-1 & L-2 job deps
* Fix report-wp-latest
* Fix failing api test
* Make get-wp-versions quicker
* Publish report immediately after test
* Test reporting in e2e-update-wc
* Fix missing parameter
* Fix env_desc, re-enable other jobs
* Enable all e2e tests
* Minor job name change
* Fix flaky test
* Add php version testing
* stringify php versions
* Re-enable all e2e tests
* Up timeout to 2min
* Remove PHP 8.0
* Add missing conditionals
* Fix php version verification script
* Fix starting dir
* Fix flakiness
* Skip e2e if api failed
* Verify woocommerce.zip early
* Add token
* Delete test summary on github for the meantime
* Use default playwright config
* More meaningful variable names
* Update step titles based on review
* Use expect.poll()
* Minor spacing corrections
* Use `stable-check` endpoint, delete unnecessary loop
* Update locators to be JN-compatible
* Fix erroneous getting of release tag
* Fix conflict of "No thanks" button locator with that of WP Mail Logging's
* Update github-script action to v6
* Revert to 'Cancelled'
* Remove unnecessary step
* Provide missing env variables
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Jonathan Lane <lanej0@users.noreply.github.com>
* Add `maxFailures`
* Intentionally fail a test for testing the failure limit
* Add changelog
* Add step to get total number of E2E tests
* Increase max failures
* Add warnings to the test summary.
Also moved some stuff around.
* Update changelog
* Use env var to set `maxFailures` value instead of hard-coding it
* Make sure env var is converted to a number
* Allow maxFailures even if not on CI
* Show incomplete e2e run warning only when E2E_MAX_FAILURES is set
* Safely escape new lines in comment body
* Fix returned stringified summary
* Remove unnecessary newline escaping
* Fix incorrect reference to E2E_MAX_FAILURES
* Minor formatting change
* Shorten warning messages
* Minor change in warning message
* Intentionally fail only 1 test
* Remove intentional failures
* Removed duplicate env declaration
* Correct pnpm command and playwright config file location
* Simplify reference to CI variable
* Remove unnecessary if condition
* Remove references to the old E2E_PLAYWRIGHT flag
* Remove references to the old E2E_PPTR_SUMMARY_PATH variable
* Test for 2 warnings
* Revert to correct locator
* Test for only 1 warning
* Pass all tests
Vladimir Reznichenko
Bart Kalisz
Adrian Moldovan
Sam Seay
nigeljamesstevenson
Adrian Moldovan
Ron Rennick
Christopher Allford
Ron Rennick
Rodel Calasagsag
Jeffrey Paul
Paul Sealock
jonathansadowski
Jamel Noel Reid