Commit Graph

133 Commits

Author SHA1 Message Date
Joshua T Flowers 87134be331
Run performance metrics on pull request (#52510)
* Run performance metrics on pull request

* Add delta checks to performance scripts

* Bump sample size to 11 for metric tests

* Reduce performance threshold for frontend to 5%

* Make metric tests required

* Remove potentially flaky cumulative layout shift metric

* Add changelog entry

* Performance metrics: minor tweaks.

---------

Co-authored-by: Vladimir Reznichenko <kalessil@gmail.com>
2024-11-12 17:35:08 -05:00
Vladimir Reznichenko 02edbf101d
[dev] Monorepo: git-checkout tweaks in project-jobs. (#52534) 2024-11-04 12:28:37 +01:00
Vladimir Reznichenko e47384d16b
[dev] Monorepo: consolidate ci-workflow (include changelog/md-files validation) (#52398) 2024-11-04 10:01:57 +01:00
Vladimir Reznichenko ed7d8c8505
[dev] CI: reduce number of wp-env startup crashes (take 2) (#52254)
To reduce wp-env startup crashes in CI, bump the ut's version to the latest and downgrade the MariaDB version used under the hood to 11.2. Related to https://github.com/WordPress/gutenberg/issues/62242, since it's addressed, we are good to remove the patch.
2024-10-23 11:13:20 +02:00
Vladimir Reznichenko e914c862a0
[dev] CI: revisit ci-workflow skip strategy. (#51876)
In this PR, we update the skip strategy for ci-workflow to use the GitHub actions feature and simply filter out the changed paths (skip docs, changes).
2024-10-03 11:11:27 +02:00
Jonathan Lane 4f3e310dcd
Add a retry on the setup test environment step (#51871)
* Add a retry on the setup test environment step

* Minor change to an e2e test to trigger CI

* Add changefile(s) from automation for the following project(s): woocommerce

---------

Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: github-actions <github-actions@github.com>
2024-10-03 00:53:53 +02:00
Vladimir Reznichenko 6b5dbbc6ac
[dev] Fix for an error introduced in #51845 (#51847) 2024-10-02 11:00:16 +02:00
Vladimir Reznichenko d26816d41f
[dev] CI: skip linting and testing jobs when merging changelog deletion PRs into trunk (#51845) 2024-10-02 10:52:06 +02:00
Vladimir Reznichenko ce5f7cb471
[dev] CI: drop baseline caching in perfromance metrics job (#51804) 2024-10-01 09:05:53 +02:00
Vladimir Reznichenko d3b0f153ad
[dev] CI: feedback on updated perfromance metrics job (#51772) 2024-09-30 16:39:16 +02:00
Vladimir Reznichenko bc8067a1ce
[dev] CI: new version of perfromance metrics job (take 2) (#51710)
In this PR, we are re-iterating the job speedup and leveraging single wp-instance, incremental builds, and baseline reports caching. The original benchmarking tool is still available but bypassed as we evaluate results for this iteration approach.
2024-09-30 09:57:52 +02:00
Adrian Moldovan cfdaa7e6cb
[testing workflows] Fix undefined BuildKite Analytics message (#50641) 2024-08-16 09:50:03 -07:00
Adrian Moldovan ff347bdcd4
[testing workflow] Disable shards when re-running failed tests (#50492) 2024-08-09 13:49:19 +03:00
Adrian Moldovan a3161244b3
[test workflows] Add LAST_FAILED_RUN switch (#50452) 2024-08-07 21:57:36 +03:00
Adrian Moldovan 81e5d24d3d
[testing workflows] Fix artifacts path for blocks e2e (#50432) 2024-08-07 11:08:30 +03:00
Adrian Moldovan adbdcceb28
[testing workflows] Only re-run failed e2e tests (#50267)
* Use Playwright's last info run

* Add changelog

* Fix workflow syntax

* Fix workflow syntax

* Try with continue-on-error

* Try with merge-multiple

* Try with pattern instead of name

* Print the run command

* Print the run command

* Use LAST_RUN_FLAG

* Update outputDir

* Update upload paths and parse the file content

* Don't try to download lat run info if the test type is not e2e

* Rename the downloaded file

* Rename the downloaded file

* Rename the downloaded file

* Full path to the downloaded file

* Check download path content

* Check download path content

* Check the last-run file in the right path

* Fail different tests on different run attempts

* Fix artifact name

* Force test failures

* Cleanup console logs

* Cleanup console logs

* Cleanup console logs

* Log if last run info exists but no tests are found

* Revert test changes
2024-08-06 14:50:25 -07:00
Adrian Moldovan e9146f5ba4
[testing workflows] Update concurrency group for CI workflow (#50407) 2024-08-06 15:18:56 +02:00
Adrian Moldovan 657a3fb99a
[testing workflows] Update concurrency to not cancel the workflow when running on trunk (#50397) 2024-08-06 13:39:57 +03:00
Bart Kalisz 52119dfcc9
Fix CI Metrics job (#50214) 2024-08-06 11:23:04 +02:00
Vladimir Reznichenko 559c91626c
CI: minor speed boost for Check Asset Sizes and Build Project Jobs. (#50342) 2024-08-05 12:58:39 +02:00
Vladimir Reznichenko 8da5a91842
CI: speedup 'Build Project Jobs' (#50197)
In this PR, we use shallow checkout if the environment provides no base_ref, which is needed for identifying changed files. If no ref provided, we are not able to calculate the diff and therefore go with faster shallow copy.
2024-08-01 14:18:08 +02:00
Adrian Moldovan 0ba308247a
[testing workflows] Add pre-release checks (#50204) 2024-08-01 13:48:58 +02:00
Adrian Moldovan 017fe40928
[testing workflows] Use choices for on-demand test run input (#50170) 2024-07-31 10:48:14 -07:00
Vladimir Reznichenko 045919d107
CI: git checkout speed tweaks in linting jobs. (#50169) 2024-07-31 14:31:41 +02:00
Vladimir Reznichenko 4b90743ada
CI: add some comments to CI-workflow about intended jobs generation strategy. (#50114) 2024-07-31 08:46:55 +02:00
Adrian Moldovan c12cb6ccdf
[tests] Move API core tests as a suite in e2e-pw (#50024) 2024-07-29 13:51:36 +03:00
Adrian Moldovan f06302bbb5
[testing workflows] Harden the run condition for reporting jobs to only run in the monorepo (#50017) 2024-07-26 12:40:14 +01:00
Vladimir Reznichenko 2d6811e689
CI: speedup 'Check Asset Sizes' job (take 2) (#49992) 2024-07-26 11:21:58 +02:00
Vladimir Reznichenko b28ee008c6
CI: improve flacky test reporting job execution time (#49665)
In this PR we use pre-build version of flaky jobs reporter, in order to speed up the job.
2024-07-22 16:33:31 +02:00
Vladimir Reznichenko 419a538901
CI: fixes for handling workflow cancellation. (#49709) 2024-07-19 12:32:10 +02:00
Vladimir Reznichenko 444f105eba
CI: Re-group PHPUnit jobs. (#49443)
In this PR, we reduce build time for REST API tests (lightweight build) and regroup a bit of PHPUnit jobs (fewer jobs with the same coverage).
2024-07-15 13:39:07 +02:00
Adrian Moldovan b9d6439b9d
Revert "Update ci.yml to be dispatchable and skippable" (#49442) 2024-07-12 12:52:16 +03:00
Adrian Moldovan e4aff57e56
Fix path to wp-env.json file in release-checks runs (#49397) 2024-07-11 20:27:35 +03:00
Adrian Moldovan 3ff155182e
Fix Blocks e2e environment setup in release checks (#49379) 2024-07-11 15:45:30 +03:00
Vladimir Reznichenko 4311640de2
Monorepo: caching deps per target package in GH actions (#49020)
In this PR, we are implementing per-package build and caching, which optimizes build and fetching cached dependencies times across our workflows.
2024-07-10 14:39:49 +02:00
jonathansadowski b8c2cf8cde
Update ci.yml to be dispatchable and skippable (#45331)
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
2024-07-09 16:50:57 +03:00
Vladimir Reznichenko 68532ce433
CI: update linting jobs to skip build step (#49193)
In this PR, we introduce changes to linting and PHPUnit jobs in order to speed them up: skip or minimize the build step.
2024-07-08 08:39:39 +02:00
Adrian Moldovan cef644078e
Remove the matrix.testEnv condition from the build step (#49149) 2024-07-04 14:50:49 +03:00
Adrian Moldovan 62c74d0ed4
[e2e tests] Add support for external sites in CI (#49017) 2024-07-04 12:15:43 +03:00
Vladimir Reznichenko f8c95a8db8
CI: caching tune (part one - relocate playwright caching) (#49081)
In this PR we ar relocating playwright downloads caching into the monorepo action (behind input parameter). That consolidates monorepo-related caching and also reduces the time fetching the cache action in external workflows.
2024-07-04 06:55:24 +02:00
Adrian Moldovan 36c28b6ad2
Move buildkite-test-collector to devDependencies (#49051) 2024-07-03 11:31:40 +03:00
Adrian Moldovan 94b93d40ff
[testing workflows] Removed deprecated reporting CI steps (#49025)
* Removed deprecated reporting steps

* Test change

* Revert test change
2024-07-02 15:40:05 -03:00
Vladimir Reznichenko 6b121e2f34
CI: caching tweaks in GitHub actions (#48865)
* Tooling: cache pnp deps

* Tooling: try fixing cache errors

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: added pre-build steps to solve concurrency issues

* Tooling: limit workers

* Tooling: revert to original build command

* Tooling: playwright related tweaks

* Tooling: minor tweaks.

* Tooling: fix caching path.

* Tooling: trigger php builds to test deps caching.

* Tooling: trigger php builds to test deps caching.

* Tooling: lock caching to monorepo setup, so it doesn't slow down build steps.

* Tooling: lock Playwright cache to build, as build is pre-requisite for running tests.

* Tooling: added caching build-folders.

* Tooling: added caching build-folders.

* Tooling: added caching build-folders.

* Revert "Tooling: added caching build-folders."

This reverts commit c57b9dff13.

* Tooling: added caching build-folders.

* Tooling: cleanup sweep.

* Tooling: cleanup sweep.

* Tooling: added a changelog entry.

* Tooling: cleanup.

* Tooling: try speedup live branches build.

* Tooling: try speedup live branches build.

* Tooling: cleanup.

* Tooling: try speedup live branches build.

* Tooling: try speedup live branches build.
2024-06-28 14:50:32 +01:00
Adrian Moldovan b098a393b8
[test reports] Optimize the reporting job - don't duplicate reports and remove Allure report (#48612)
* * Use run_attempt in the artifact name

* Remove the Allure report steps

* Test change to trigger tests

* Only run the publish report step if there is an artifact-id

* Only run the publish report step if there is an artifact-id

* Revert test changes

* Also condition the deprecated publish report step
2024-06-19 09:37:50 -07:00
Adrian Moldovan f2fd4968f8
[ci] Always run the resolve artifacts path (#48508) 2024-06-17 10:16:04 -07:00
Adrian Moldovan 3fb5d40a43
[test reports] Send the run attempt to the reports workflow (#48519) 2024-06-17 10:14:32 -07:00
Adrian Moldovan 90398903ab
Fix core api tests results path and artifacts naming pattern (#48490)
* Fix api tests results path

* Add changelog

* Run Core API tests - HPOS disabled on pull_request

* Update the artifacts name pattern to avoid conflicts

* Revert pull_request trigger
2024-06-14 09:03:43 -07:00
Adrian Moldovan 1f534d0280
[e2e tests] Update the report configuration for core e2e tests and add a new workflow call for report publishing (#48424)
* Add a call to a new report workflow in woocommerce-test-reports repo

* Test change to trigger tests

* Update report title

* Test change to trigger tests

* Check REPORT_TITLE

* Add quotes on all arguments

* Update reporting configuration to all core e2e jobs

* Add changelog

* Use GITHUB_HEAD_REF for pull_request

* Set REPORT_TITLE for workflow dispatch

* Updated trigger from daily-e2e to daily-checks
2024-06-12 13:38:37 -07:00
Jonathan Lane e5e51a41b8
Update Playwright to 1.44.1 from 1.41.1 (#48291)
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
2024-06-11 21:52:48 +03:00
Adrian Moldovan f3a7c0a6ec
[ci] Skip build for lint jobs (#48234) 2024-06-11 20:01:43 +03:00