In this PR, we are re-iterating the job speedup and leveraging single wp-instance, incremental builds, and baseline reports caching. The original benchmarking tool is still available but bypassed as we evaluate results for this iteration approach.
* Include performance tests job in the needed jobs for evaluation
* Use Github api to get the workflow jobs status
* Evaluate project-jobs
* See results
* Check for cancelled jobs
* Add evaluate-project-jobs.js and rename optional jobs
* Fix script path
* Checkout first
* And then fix the path again.
* Update env variable name and be more detailed in what are the required variables
* Use a flat matrix env variable
* Update to trigger test jobs
* Use the jobs name to evaluate optional requirement
* Fix conditions
* Prettier print
* Test lint job failure
* Remove unused MATRIX env variable
* Fix test
* Remove unused MATRIX variable check
* Only run my account e2e tests
* Remove unused function
* Nicer console printing
* Revert change that triggers failing lint job
* Force failing e2e test
* Revert e2e test command and forced failure
* Added test data and test mode
* Added more test data
* Fixed lint errors and warnings
* Exclude .github folder from eslintignore
* Change to trigger everything
* Revert change to trigger everything
* Move deprecated workflows to deprecated folder
* Configure new test job for metrics
* Add changelog
* Update start command
* Update start command
* Archive metrics results
* Fix script path
* Fix name in archive results condition
* Add the actual run command for pull_request event
* Run with pnpm --filter
* Test run for push event
* Revert test conditions
* Add WP_ARTIFACTS_PATH env variable
* Update path to readme.txt
* Revert github event conditions
* Add CODEVITALS_PROJECT_TOKEN env
* Set WP_ARTIFACTS_PATH on job level
* Add blocks templates in the changes list
* exclude dependabot from commmunity contributor label
* use prior art from block repo
---------
Co-authored-by: Ron Rennick <ronald.rennick@automattic.com>
This adds support for using the `pnpm utils ci-jobs` command in our `ci.yml` file. One of the bigger benefits to this change too is that we're now distributing a bundled version of the utils tool. This lets us run it without actually having to install the repo and will let us speed up any workflows that currently do.
* use latest commit of stalebot action, tweak configuration accordingly
* remove stalebot.js
* use default operations per run
---------
Co-authored-by: Ron Rennick <ronald.rennick@automattic.com>
* change reference of Catpure to Capture
Co-Authored-By: Vikram <93216400+vikrampm1@users.noreply.github.com>
* change reference of expicitly to explicitly
Co-Authored-By: Vikram <93216400+vikrampm1@users.noreply.github.com>
* change reference 'cutted' to 'cut'
* change reference 'determening' to 'determining'
* change reference 'retreive' to 'retrieve'
* change reference 'neccessary' to 'necessary'
* change reference 'Fitler' to 'Filter'
* change reference of "seperate" to "separate"
Co-Authored-By: Ankit K Gupta <ankit.himcs@gmail.com>
* change reference of "wether" to "whether"
Co-Authored-By: Sumit Bagthariya <67687255+qasumitbagthariya@users.noreply.github.com>
* change reference of "staus" to "status"
* change reference of "retrive" to "retrieve"
* change references of "gatways" to "gateways"
* change references of "existant" to "existent"
* change reference of "requries" to "requires"
* change reference of "configuation" to "configuration"
* change reference of "processsing" to "processing"
* change reference of "represenation" to "representation"
* change reference of "dimentions" to "dimensions"
* change references of "reigster" to "register"
* change reference of "colum" to "column"
* change reference of "transtions" to "transitions"
* change references of "intially" to "initially"
* change references of "orignal" to "original"
* change references of "deprected" to "deprecated"
* change references of "paramter" to "parameter"
* change reference of "intance" to "instance"
* change reference of "elemets" to "elements"
* change references of "funcitons" to "functions"
* change reference of "specificed" to "specified"
* change references of "atributes" to "attributes"
* change reference of "tast" to "task"
* change reference of "chaning" to "changing"
* change reference of "retreiving" to "retrieving"
* change reference of "caluclation" to "calculation"
* change references of "Invaid" to "Invalid"
* change references of "paramaters" to "parameters"
* change reference of "Additonal" to "Additional"
* change reference of "teh" to "the"
* change reference of "evalutes" to "evaluates"
* change reference of "addedd" to "added"
* change reference of "excempt" to "exempt"
* change reference of "sequencially" to "sequentially"
* change reference of "previos" to "previous"
* change reference of "elegible" to "eligible"
* change references of "Boostrap" to "Bootstrap"
* change references of "compability" to "compatibility"
* change reference of "heirarchy" to "hierarchy"
* change references of "visibilty" to "visibility"
* change reference of "comparsion" to "comparison"
* change reference of "capabilties" to "capabilities"
* change reference of "datatores" to "datastores"
* change reference of "occured" to "occurred"
* change reference of "coresponding" to "corresponding"
* change references of "thier" to "their"
* change reference of "sucessfully" to "successfully"
* change reference of "insde" to "inside"
* change reference of "nagivation" to "navigation"
* change references of "visiblity" to "visibility"
* change reference of "documentaiton" to "documentation"
* change reference of "anayltics" to "analytics"
* change reference of "intalling" to "installing"
* change reference of "mininum" to "minimum"
* change references of "intial" to "initial"
* change reference of "Feld" to "Field"
* change reference of "taks" to "task"
* change reference of "trasnfer" to "transfer"
* change reference of "respone" to "response"
* change reference of "Extenstions" to "Extensions"
* change reference of "detault" to "default"
* change reference of "simultanious" to "simultaneous"
* change reference of "overides" to "overrides"
* change references of "Indvidual" to "Individual"
* change reference of "refering" to "referring"
* change reference of "aginst" to "against"
* change reference of "execuatable" to "executable"
* change reference of "repsonse" to "response"
* change reference of "documention" to "documentation"
* change reference of "asumed" to "assumed"
* change reference of "Minium" to "Minimum"
* change reference of "unqiue" to "unique"
* change references of "existance" to "existence"
* change reference of "compatability" to "compatibility"
* change reference of "Taxnomy" to "Taxonomy"
* change reference of "quering" to "querying"
* change reference of "retrun" to "return"
* change reference of "informations" to "information"
Co-Authored-By: Viktor Szépe <viktor@szepe.net>
* Add changelog
* Add changelog
* Fix typo
---------
Co-authored-by: Vikram <93216400+vikrampm1@users.noreply.github.com>
Co-authored-by: Ankit K Gupta <ankit.himcs@gmail.com>
Co-authored-by: Sumit Bagthariya <67687255+qasumitbagthariya@users.noreply.github.com>
Co-authored-by: Viktor Szépe <viktor@szepe.net>
Co-authored-by: Chi-Hsuan Huang <chihsuan.tw@gmail.com>
* First pass at updating release test workflow
* Add changelog
* Set dir env variables
* Update to workflow
* Fix indent
* Fix indent
* Clean up indent
* Re-order steps
* Change order of jobs
* Added common php versions
* Update pipeline
* Update some labels
* Simplify for testing
* Update paths
* Create tmp folder
* Fix path
* Paths
* Try outputting some debugging
* Add step ID back
* Remove working directory
* Another path tweak
* Add API release tests
* Add k6 tests
* Add PHP tests
* Launch wp-env during PHP tests
* Try default values
* Tweak some settings, add WP testing
* Tweak some settings
* Re-order e2e steps
* Update step descriptions
* Reorganize tests, add plugin tests
* Enable only e2e job
* Initial set up to run against release smoke test site
* Fix syntax
* Temporarily disable update wc spec
* Temporarily disable downloading woocommerce zip
* Download release zip using tag name
* Fix wrong job name
* Fix wrong job name
* Fix dir
* Delete fetch-asset-id.js
* REfactor update-woocommerce spec
* Add error handling
* Download release zip by tag
* Refactor update woo spec to download zip by tag
* Correct job name
* fail test on invalid tag
* Enable all e2e tests
* Run api tests before e2e tests
* Fix job dependency
* Add customer credentials to api job
* Separate job for WC Update
* Combine e2e allure-results, then report
* Enable report job
* Fix context
* Change job and artifact names
* Use test s3 path
* Minor job name change
* Upload artifacts to bucket
* Correct s3 path
* Add quiet option
* Retain video on failures
* Finalize s3 path
* Try WP latest-1
* Revert to wp latest
* Refine search for woocommerce zip asset
* Get created-at
* Specify repo in gh command
* Slugify env description
* Trim space
* Sync with bucket instead of copy
* Remove invalid --recursive flag
* Re-add missing step to combine e2e results from update wc test
* Ensure artifact upload on test failure
* Enable all e2e tests on WP latest
* Retain existing data before updating WC
* Make test compatible with 'Canceled' and 'Cancelled'
* Set env_desc as env var
* Re-add deleted file
* Fix UPDATE_WC in daily smoke test workflow
* Add tracing in global setup
* Remove tracing
* Temporarily run only basic spec
* Job for WP Latest-1 & 2
* Fix "Required input 'created_at' not provided"
* Minor rename
* Remove install filter
* Install deps in get-wp-matrix
* Delete get-wp-versions.js
* Add get-wp-versions.js to e2e-pw folder
* REname file
* REfactor
* Refactor script for getting WP prev versions
* Update job dependencies
* Temporarily remove disabled jobs
* Allow e2e-wp-latest after api test failure
* Update L-1 & L-2 job deps
* Fix report-wp-latest
* Fix failing api test
* Make get-wp-versions quicker
* Publish report immediately after test
* Test reporting in e2e-update-wc
* Fix missing parameter
* Fix env_desc, re-enable other jobs
* Enable all e2e tests
* Minor job name change
* Fix flaky test
* Add php version testing
* stringify php versions
* Re-enable all e2e tests
* Up timeout to 2min
* Remove PHP 8.0
* Add missing conditionals
* Fix php version verification script
* Fix starting dir
* Fix flakiness
* Skip e2e if api failed
* Verify woocommerce.zip early
* Add token
* Delete test summary on github for the meantime
* Use default playwright config
* More meaningful variable names
* Update step titles based on review
* Use expect.poll()
* Minor spacing corrections
* Use `stable-check` endpoint, delete unnecessary loop
* Update locators to be JN-compatible
* Fix erroneous getting of release tag
* Fix conflict of "No thanks" button locator with that of WP Mail Logging's
* Update github-script action to v6
* Revert to 'Cancelled'
* Remove unnecessary step
* Provide missing env variables
---------
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Jonathan Lane <lanej0@users.noreply.github.com>
* Add `maxFailures`
* Intentionally fail a test for testing the failure limit
* Add changelog
* Add step to get total number of E2E tests
* Increase max failures
* Add warnings to the test summary.
Also moved some stuff around.
* Update changelog
* Use env var to set `maxFailures` value instead of hard-coding it
* Make sure env var is converted to a number
* Allow maxFailures even if not on CI
* Show incomplete e2e run warning only when E2E_MAX_FAILURES is set
* Safely escape new lines in comment body
* Fix returned stringified summary
* Remove unnecessary newline escaping
* Fix incorrect reference to E2E_MAX_FAILURES
* Minor formatting change
* Shorten warning messages
* Minor change in warning message
* Intentionally fail only 1 test
* Remove intentional failures
* Removed duplicate env declaration
* Correct pnpm command and playwright config file location
* Simplify reference to CI variable
* Remove unnecessary if condition
* Remove references to the old E2E_PLAYWRIGHT flag
* Remove references to the old E2E_PPTR_SUMMARY_PATH variable
* Test for 2 warnings
* Revert to correct locator
* Test for only 1 warning
* Pass all tests
* Split changelog script from code-freeze
* Remove hardcoded repository info from testing
* Remove generation of changelog.txt file from automation
* Correct comments and add back write of readme
Vladimir Reznichenko
Bart Kalisz
Adrian Moldovan
Sam Seay
nigeljamesstevenson
Adrian Moldovan
Ron Rennick
Christopher Allford
Ron Rennick
Rodel Calasagsag
Jeffrey Paul
Paul Sealock
jonathansadowski
Jamel Noel Reid