Commit Graph

121 Commits

Author SHA1 Message Date
Adrian Moldovan ff347bdcd4
[testing workflow] Disable shards when re-running failed tests (#50492) 2024-08-09 13:49:19 +03:00
Adrian Moldovan a3161244b3
[test workflows] Add LAST_FAILED_RUN switch (#50452) 2024-08-07 21:57:36 +03:00
Adrian Moldovan 81e5d24d3d
[testing workflows] Fix artifacts path for blocks e2e (#50432) 2024-08-07 11:08:30 +03:00
Adrian Moldovan adbdcceb28
[testing workflows] Only re-run failed e2e tests (#50267)
* Use Playwright's last info run

* Add changelog

* Fix workflow syntax

* Fix workflow syntax

* Try with continue-on-error

* Try with merge-multiple

* Try with pattern instead of name

* Print the run command

* Print the run command

* Use LAST_RUN_FLAG

* Update outputDir

* Update upload paths and parse the file content

* Don't try to download lat run info if the test type is not e2e

* Rename the downloaded file

* Rename the downloaded file

* Rename the downloaded file

* Full path to the downloaded file

* Check download path content

* Check download path content

* Check the last-run file in the right path

* Fail different tests on different run attempts

* Fix artifact name

* Force test failures

* Cleanup console logs

* Cleanup console logs

* Cleanup console logs

* Log if last run info exists but no tests are found

* Revert test changes
2024-08-06 14:50:25 -07:00
Adrian Moldovan e9146f5ba4
[testing workflows] Update concurrency group for CI workflow (#50407) 2024-08-06 15:18:56 +02:00
Adrian Moldovan 657a3fb99a
[testing workflows] Update concurrency to not cancel the workflow when running on trunk (#50397) 2024-08-06 13:39:57 +03:00
Bart Kalisz 52119dfcc9
Fix CI Metrics job (#50214) 2024-08-06 11:23:04 +02:00
Vladimir Reznichenko 559c91626c
CI: minor speed boost for Check Asset Sizes and Build Project Jobs. (#50342) 2024-08-05 12:58:39 +02:00
Vladimir Reznichenko 8da5a91842
CI: speedup 'Build Project Jobs' (#50197)
In this PR, we use shallow checkout if the environment provides no base_ref, which is needed for identifying changed files. If no ref provided, we are not able to calculate the diff and therefore go with faster shallow copy.
2024-08-01 14:18:08 +02:00
Adrian Moldovan 0ba308247a
[testing workflows] Add pre-release checks (#50204) 2024-08-01 13:48:58 +02:00
Adrian Moldovan 017fe40928
[testing workflows] Use choices for on-demand test run input (#50170) 2024-07-31 10:48:14 -07:00
Vladimir Reznichenko 045919d107
CI: git checkout speed tweaks in linting jobs. (#50169) 2024-07-31 14:31:41 +02:00
Vladimir Reznichenko 4b90743ada
CI: add some comments to CI-workflow about intended jobs generation strategy. (#50114) 2024-07-31 08:46:55 +02:00
Adrian Moldovan c12cb6ccdf
[tests] Move API core tests as a suite in e2e-pw (#50024) 2024-07-29 13:51:36 +03:00
Adrian Moldovan f06302bbb5
[testing workflows] Harden the run condition for reporting jobs to only run in the monorepo (#50017) 2024-07-26 12:40:14 +01:00
Vladimir Reznichenko 2d6811e689
CI: speedup 'Check Asset Sizes' job (take 2) (#49992) 2024-07-26 11:21:58 +02:00
Vladimir Reznichenko b28ee008c6
CI: improve flacky test reporting job execution time (#49665)
In this PR we use pre-build version of flaky jobs reporter, in order to speed up the job.
2024-07-22 16:33:31 +02:00
Vladimir Reznichenko 419a538901
CI: fixes for handling workflow cancellation. (#49709) 2024-07-19 12:32:10 +02:00
Vladimir Reznichenko 444f105eba
CI: Re-group PHPUnit jobs. (#49443)
In this PR, we reduce build time for REST API tests (lightweight build) and regroup a bit of PHPUnit jobs (fewer jobs with the same coverage).
2024-07-15 13:39:07 +02:00
Adrian Moldovan b9d6439b9d
Revert "Update ci.yml to be dispatchable and skippable" (#49442) 2024-07-12 12:52:16 +03:00
Adrian Moldovan e4aff57e56
Fix path to wp-env.json file in release-checks runs (#49397) 2024-07-11 20:27:35 +03:00
Adrian Moldovan 3ff155182e
Fix Blocks e2e environment setup in release checks (#49379) 2024-07-11 15:45:30 +03:00
Vladimir Reznichenko 4311640de2
Monorepo: caching deps per target package in GH actions (#49020)
In this PR, we are implementing per-package build and caching, which optimizes build and fetching cached dependencies times across our workflows.
2024-07-10 14:39:49 +02:00
jonathansadowski b8c2cf8cde
Update ci.yml to be dispatchable and skippable (#45331)
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
2024-07-09 16:50:57 +03:00
Vladimir Reznichenko 68532ce433
CI: update linting jobs to skip build step (#49193)
In this PR, we introduce changes to linting and PHPUnit jobs in order to speed them up: skip or minimize the build step.
2024-07-08 08:39:39 +02:00
Adrian Moldovan cef644078e
Remove the matrix.testEnv condition from the build step (#49149) 2024-07-04 14:50:49 +03:00
Adrian Moldovan 62c74d0ed4
[e2e tests] Add support for external sites in CI (#49017) 2024-07-04 12:15:43 +03:00
Vladimir Reznichenko f8c95a8db8
CI: caching tune (part one - relocate playwright caching) (#49081)
In this PR we ar relocating playwright downloads caching into the monorepo action (behind input parameter). That consolidates monorepo-related caching and also reduces the time fetching the cache action in external workflows.
2024-07-04 06:55:24 +02:00
Adrian Moldovan 36c28b6ad2
Move buildkite-test-collector to devDependencies (#49051) 2024-07-03 11:31:40 +03:00
Adrian Moldovan 94b93d40ff
[testing workflows] Removed deprecated reporting CI steps (#49025)
* Removed deprecated reporting steps

* Test change

* Revert test change
2024-07-02 15:40:05 -03:00
Vladimir Reznichenko 6b121e2f34
CI: caching tweaks in GitHub actions (#48865)
* Tooling: cache pnp deps

* Tooling: try fixing cache errors

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: added pre-build steps to solve concurrency issues

* Tooling: limit workers

* Tooling: revert to original build command

* Tooling: playwright related tweaks

* Tooling: minor tweaks.

* Tooling: fix caching path.

* Tooling: trigger php builds to test deps caching.

* Tooling: trigger php builds to test deps caching.

* Tooling: lock caching to monorepo setup, so it doesn't slow down build steps.

* Tooling: lock Playwright cache to build, as build is pre-requisite for running tests.

* Tooling: added caching build-folders.

* Tooling: added caching build-folders.

* Tooling: added caching build-folders.

* Revert "Tooling: added caching build-folders."

This reverts commit c57b9dff13.

* Tooling: added caching build-folders.

* Tooling: cleanup sweep.

* Tooling: cleanup sweep.

* Tooling: added a changelog entry.

* Tooling: cleanup.

* Tooling: try speedup live branches build.

* Tooling: try speedup live branches build.

* Tooling: cleanup.

* Tooling: try speedup live branches build.

* Tooling: try speedup live branches build.
2024-06-28 14:50:32 +01:00
Adrian Moldovan b098a393b8
[test reports] Optimize the reporting job - don't duplicate reports and remove Allure report (#48612)
* * Use run_attempt in the artifact name

* Remove the Allure report steps

* Test change to trigger tests

* Only run the publish report step if there is an artifact-id

* Only run the publish report step if there is an artifact-id

* Revert test changes

* Also condition the deprecated publish report step
2024-06-19 09:37:50 -07:00
Adrian Moldovan f2fd4968f8
[ci] Always run the resolve artifacts path (#48508) 2024-06-17 10:16:04 -07:00
Adrian Moldovan 3fb5d40a43
[test reports] Send the run attempt to the reports workflow (#48519) 2024-06-17 10:14:32 -07:00
Adrian Moldovan 90398903ab
Fix core api tests results path and artifacts naming pattern (#48490)
* Fix api tests results path

* Add changelog

* Run Core API tests - HPOS disabled on pull_request

* Update the artifacts name pattern to avoid conflicts

* Revert pull_request trigger
2024-06-14 09:03:43 -07:00
Adrian Moldovan 1f534d0280
[e2e tests] Update the report configuration for core e2e tests and add a new workflow call for report publishing (#48424)
* Add a call to a new report workflow in woocommerce-test-reports repo

* Test change to trigger tests

* Update report title

* Test change to trigger tests

* Check REPORT_TITLE

* Add quotes on all arguments

* Update reporting configuration to all core e2e jobs

* Add changelog

* Use GITHUB_HEAD_REF for pull_request

* Set REPORT_TITLE for workflow dispatch

* Updated trigger from daily-e2e to daily-checks
2024-06-12 13:38:37 -07:00
Jonathan Lane e5e51a41b8
Update Playwright to 1.44.1 from 1.41.1 (#48291)
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
2024-06-11 21:52:48 +03:00
Adrian Moldovan f3a7c0a6ec
[ci] Skip build for lint jobs (#48234) 2024-06-11 20:01:43 +03:00
Adrian Moldovan a7f634d5ef
[blocks e2e] Move Blocks e2e into ci.yml (#48224) 2024-06-11 19:57:23 +03:00
Adrian Moldovan 1533afdca3
[e2e tests] Fix e2e test results path and report name (#48320)
* Fix the report name sent to woocommerce-test-reports workflow

* Add test type

* Test change

* Remove test type

* Merge artifacts to keep allure-results

* Overwrite artifacts if one already exists with the same name

* Test changes to trigger tests

* Fix allure-results and .state paths

* Filter tests for faster feedback

* Debug artifacts content

* Use name instead of pattern

* Use name instead of pattern

* Don't use an intermediate path

* Use artifact name and path instead of only pattern

* Use artifact name and path instead of only pattern

* Use artifact name and path instead of only pattern

* Remove testMatch filter to run all tests

* Revert test change

* Add changelog
2024-06-10 11:21:46 -07:00
Adrian Moldovan 8fc6970b90
[ci] Merge test jobs into a single generic job (#48175) 2024-06-06 21:50:52 +03:00
Adrian Moldovan 26158ee6b4
[testing workflows] Replace push to tags trigger with release published and updated (#48209)
Update push to tag triggers with release trigger
2024-06-06 09:51:02 -07:00
Adrian Moldovan 7fb40ffc0b
[testing workflows] Migrate release checks to ci.yml (#48113)
* Deprecate pr-smoke-test.yml

* Add configs for each environment

* Add scripts

* Add arguments to playwright command

* Update pnpm scripts and existing e2e projects

* Fix shellcheck warning

* Update test env start for Gutenberg stable run

* Add changelog

* Remove unused var

* Set the USE_WP_ENV var

* Configure daily e2e jobs

* Rename some steps

* Add release checks jobs and deprecate  smoke-test-release.yml

* Deprecate  smoke-test-release.yml

* Add tests-release-checks.yml to trigger ci.yml on push to tags

* Add the pull_request trigger to test the workflow

* Remove the pull_request trigger to test the workflow

* Add the pull_request trigger to test the workflow

* Add the tag input in ci.yml

* Remove import and use core reference

* Only build if build-plugin is true

* Use release-artifact instead

* Split setup-monorepo step

* Use tag: 'nightly' to test

* Use inputs.tag directly in the wp-env setup step

* Add script to override the wp-env config

* Run the env config script in ci

* Set the artifact name

* Fix workflow syntax

* Add changelog

* Validate RELEASE_TAG and ARTIFACT_NAME

* Use real ref type and name

* Use real ref type and name

* Use real ref type and name

* Use github context instead

* Test with 9.0.0-beta.2

* Add push to tags trigger directly in ci.yml

* Configure one job to run with nightly-checks

* Remove empty lines

* Format events array one element per line

* Format events array one element per line
2024-06-05 13:02:54 -03:00
Jonathan Lane e8e1391fa0
Add annotation for test PR e2e test run results (#48029)
* Add annotation for test results

* Minor change to trigger e2e tests

* Fix command, add emoji

* Update emoji

* Add changelog

* Update from PR feedback

* Revert "Add changelog"

This reverts commit 03656848eb.

* Revert "Minor change to trigger e2e tests"

This reverts commit 64e3a87b2d.

---------

Co-authored-by: Jon Lane <jon.lane@automattic.com>
2024-06-04 13:03:38 -07:00
Adrian Moldovan 9b251ce318
[testing workflows] Add test environments and deprecate the daily e2e workflow (#48101)
* Deprecate pr-smoke-test.yml

* Add configs for each environment

* Add scripts

* Add arguments to playwright command

* Update pnpm scripts and existing e2e projects

* Fix shellcheck warning

* Update test env start for Gutenberg stable run

* Add changelog

* Remove unused var

* Set the USE_WP_ENV var

* Configure daily e2e jobs

* Rename some steps
2024-06-03 14:12:45 -07:00
Adrian Moldovan 97e7f86a15
Add the slack-test-report util (#47805)
* Add the slack-test-report util

* Error if the required GitHub context variables are not set

* Error if the required SLACK_CHANNEL env var is not set

* Use a more generic message to be able to report on jobs that are not tests, like build

* Add the alert-on-failure job in ci

* Run for pull_request to test the job

* Setup monorepo first

* Force a linting error

* Better logging

* Use inputs.trigger as report name if set

* Force an e2e test failure

* Set the commit message in the notification

* Use INPUT_TRIGGER for all notifications

* Revert test changes
2024-05-24 13:18:54 -07:00
Adrian Moldovan 835625c329
Fix reporting step for daily e2e runs (#47764)
* Add condition on github.event_name for lint jobs

* Update a file to trigger a lint job

* Update condition to run only on pull_request

* Revert test change

* Always run the Slack notification step

* Add pull_request test trigger

* Update report step

* Remove branch filter

* Try with push event and branch filter

* Pass secrets to ci.yml

* Update testIgnore pattern for Gutenberg project

* Add changelog

* Remove the test push trigger

* One more test run

* Removed push test trigger
2024-05-23 09:12:10 -07:00
Adrian Moldovan f1489892aa
[ci] Only run lint jobs on pull_request event (#47652) 2024-05-23 07:16:07 -07:00
Adrian Moldovan 13ab9007f6
[testing workflows] Add workflow_call trigger in CI workflow - support daily and other types of runs (#47612)
* Add workflow_call event

* Add tests-daily-run.yml

* Remove workflow version

* Update input value

* Remove secrets from caller workflow

* Rename job

* Check context

* Check context

* Check workflow_call inputs

* Override github.event_name with inputs.trigger

* Fix input name

* Add config for daily e2e tests with Gutenberg

* Update env commands for Gutenberg run

* Add disableHpos env variable in ci-utils

* Revert the trigger to push

* Revert the env variable setting in ci-jobs

* Revert the env variable setting in ci-jobs

* Rebuild utils

* Update test env start command

* Define the Gutenberg Playwright project

* Use path relative to workspace root

* Update events for test projects

* Update events for test projects

* Add changelogs

* Remove the e2e tests with Gutenberg tests from daily runs

* Set the schedule and workflow_dispatch triggers

* Add --project default flag to only run the default tests in CI

* Renamed snapshots to match the new Playwright project name
2024-05-22 11:04:44 -07:00
Adrian Moldovan 0522511a72
[testing worklflows] Consolidate the Slack channel used for test reports (#47611) 2024-05-22 11:01:51 -03:00