Commit Graph

105 Commits

Author SHA1 Message Date
Vladimir Reznichenko b28ee008c6
CI: improve flacky test reporting job execution time (#49665)
In this PR we use pre-build version of flaky jobs reporter, in order to speed up the job.
2024-07-22 16:33:31 +02:00
Vladimir Reznichenko 419a538901
CI: fixes for handling workflow cancellation. (#49709) 2024-07-19 12:32:10 +02:00
Vladimir Reznichenko 444f105eba
CI: Re-group PHPUnit jobs. (#49443)
In this PR, we reduce build time for REST API tests (lightweight build) and regroup a bit of PHPUnit jobs (fewer jobs with the same coverage).
2024-07-15 13:39:07 +02:00
Adrian Moldovan b9d6439b9d
Revert "Update ci.yml to be dispatchable and skippable" (#49442) 2024-07-12 12:52:16 +03:00
Adrian Moldovan e4aff57e56
Fix path to wp-env.json file in release-checks runs (#49397) 2024-07-11 20:27:35 +03:00
Adrian Moldovan 3ff155182e
Fix Blocks e2e environment setup in release checks (#49379) 2024-07-11 15:45:30 +03:00
Vladimir Reznichenko 4311640de2
Monorepo: caching deps per target package in GH actions (#49020)
In this PR, we are implementing per-package build and caching, which optimizes build and fetching cached dependencies times across our workflows.
2024-07-10 14:39:49 +02:00
jonathansadowski b8c2cf8cde
Update ci.yml to be dispatchable and skippable (#45331)
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
2024-07-09 16:50:57 +03:00
Vladimir Reznichenko 68532ce433
CI: update linting jobs to skip build step (#49193)
In this PR, we introduce changes to linting and PHPUnit jobs in order to speed them up: skip or minimize the build step.
2024-07-08 08:39:39 +02:00
Adrian Moldovan cef644078e
Remove the matrix.testEnv condition from the build step (#49149) 2024-07-04 14:50:49 +03:00
Adrian Moldovan 62c74d0ed4
[e2e tests] Add support for external sites in CI (#49017) 2024-07-04 12:15:43 +03:00
Vladimir Reznichenko f8c95a8db8
CI: caching tune (part one - relocate playwright caching) (#49081)
In this PR we ar relocating playwright downloads caching into the monorepo action (behind input parameter). That consolidates monorepo-related caching and also reduces the time fetching the cache action in external workflows.
2024-07-04 06:55:24 +02:00
Adrian Moldovan 36c28b6ad2
Move buildkite-test-collector to devDependencies (#49051) 2024-07-03 11:31:40 +03:00
Adrian Moldovan 94b93d40ff
[testing workflows] Removed deprecated reporting CI steps (#49025)
* Removed deprecated reporting steps

* Test change

* Revert test change
2024-07-02 15:40:05 -03:00
Vladimir Reznichenko 6b121e2f34
CI: caching tweaks in GitHub actions (#48865)
* Tooling: cache pnp deps

* Tooling: try fixing cache errors

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: try to parallelize js builds a bit

* Tooling: added pre-build steps to solve concurrency issues

* Tooling: limit workers

* Tooling: revert to original build command

* Tooling: playwright related tweaks

* Tooling: minor tweaks.

* Tooling: fix caching path.

* Tooling: trigger php builds to test deps caching.

* Tooling: trigger php builds to test deps caching.

* Tooling: lock caching to monorepo setup, so it doesn't slow down build steps.

* Tooling: lock Playwright cache to build, as build is pre-requisite for running tests.

* Tooling: added caching build-folders.

* Tooling: added caching build-folders.

* Tooling: added caching build-folders.

* Revert "Tooling: added caching build-folders."

This reverts commit c57b9dff13.

* Tooling: added caching build-folders.

* Tooling: cleanup sweep.

* Tooling: cleanup sweep.

* Tooling: added a changelog entry.

* Tooling: cleanup.

* Tooling: try speedup live branches build.

* Tooling: try speedup live branches build.

* Tooling: cleanup.

* Tooling: try speedup live branches build.

* Tooling: try speedup live branches build.
2024-06-28 14:50:32 +01:00
Adrian Moldovan b098a393b8
[test reports] Optimize the reporting job - don't duplicate reports and remove Allure report (#48612)
* * Use run_attempt in the artifact name

* Remove the Allure report steps

* Test change to trigger tests

* Only run the publish report step if there is an artifact-id

* Only run the publish report step if there is an artifact-id

* Revert test changes

* Also condition the deprecated publish report step
2024-06-19 09:37:50 -07:00
Adrian Moldovan f2fd4968f8
[ci] Always run the resolve artifacts path (#48508) 2024-06-17 10:16:04 -07:00
Adrian Moldovan 3fb5d40a43
[test reports] Send the run attempt to the reports workflow (#48519) 2024-06-17 10:14:32 -07:00
Adrian Moldovan 90398903ab
Fix core api tests results path and artifacts naming pattern (#48490)
* Fix api tests results path

* Add changelog

* Run Core API tests - HPOS disabled on pull_request

* Update the artifacts name pattern to avoid conflicts

* Revert pull_request trigger
2024-06-14 09:03:43 -07:00
Adrian Moldovan 1f534d0280
[e2e tests] Update the report configuration for core e2e tests and add a new workflow call for report publishing (#48424)
* Add a call to a new report workflow in woocommerce-test-reports repo

* Test change to trigger tests

* Update report title

* Test change to trigger tests

* Check REPORT_TITLE

* Add quotes on all arguments

* Update reporting configuration to all core e2e jobs

* Add changelog

* Use GITHUB_HEAD_REF for pull_request

* Set REPORT_TITLE for workflow dispatch

* Updated trigger from daily-e2e to daily-checks
2024-06-12 13:38:37 -07:00
Jonathan Lane e5e51a41b8
Update Playwright to 1.44.1 from 1.41.1 (#48291)
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Bart Kalisz <bartlomiej.kalisz@gmail.com>
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
2024-06-11 21:52:48 +03:00
Adrian Moldovan f3a7c0a6ec
[ci] Skip build for lint jobs (#48234) 2024-06-11 20:01:43 +03:00
Adrian Moldovan a7f634d5ef
[blocks e2e] Move Blocks e2e into ci.yml (#48224) 2024-06-11 19:57:23 +03:00
Adrian Moldovan 1533afdca3
[e2e tests] Fix e2e test results path and report name (#48320)
* Fix the report name sent to woocommerce-test-reports workflow

* Add test type

* Test change

* Remove test type

* Merge artifacts to keep allure-results

* Overwrite artifacts if one already exists with the same name

* Test changes to trigger tests

* Fix allure-results and .state paths

* Filter tests for faster feedback

* Debug artifacts content

* Use name instead of pattern

* Use name instead of pattern

* Don't use an intermediate path

* Use artifact name and path instead of only pattern

* Use artifact name and path instead of only pattern

* Use artifact name and path instead of only pattern

* Remove testMatch filter to run all tests

* Revert test change

* Add changelog
2024-06-10 11:21:46 -07:00
Adrian Moldovan 8fc6970b90
[ci] Merge test jobs into a single generic job (#48175) 2024-06-06 21:50:52 +03:00
Adrian Moldovan 26158ee6b4
[testing workflows] Replace push to tags trigger with release published and updated (#48209)
Update push to tag triggers with release trigger
2024-06-06 09:51:02 -07:00
Adrian Moldovan 7fb40ffc0b
[testing workflows] Migrate release checks to ci.yml (#48113)
* Deprecate pr-smoke-test.yml

* Add configs for each environment

* Add scripts

* Add arguments to playwright command

* Update pnpm scripts and existing e2e projects

* Fix shellcheck warning

* Update test env start for Gutenberg stable run

* Add changelog

* Remove unused var

* Set the USE_WP_ENV var

* Configure daily e2e jobs

* Rename some steps

* Add release checks jobs and deprecate  smoke-test-release.yml

* Deprecate  smoke-test-release.yml

* Add tests-release-checks.yml to trigger ci.yml on push to tags

* Add the pull_request trigger to test the workflow

* Remove the pull_request trigger to test the workflow

* Add the pull_request trigger to test the workflow

* Add the tag input in ci.yml

* Remove import and use core reference

* Only build if build-plugin is true

* Use release-artifact instead

* Split setup-monorepo step

* Use tag: 'nightly' to test

* Use inputs.tag directly in the wp-env setup step

* Add script to override the wp-env config

* Run the env config script in ci

* Set the artifact name

* Fix workflow syntax

* Add changelog

* Validate RELEASE_TAG and ARTIFACT_NAME

* Use real ref type and name

* Use real ref type and name

* Use real ref type and name

* Use github context instead

* Test with 9.0.0-beta.2

* Add push to tags trigger directly in ci.yml

* Configure one job to run with nightly-checks

* Remove empty lines

* Format events array one element per line

* Format events array one element per line
2024-06-05 13:02:54 -03:00
Jonathan Lane e8e1391fa0
Add annotation for test PR e2e test run results (#48029)
* Add annotation for test results

* Minor change to trigger e2e tests

* Fix command, add emoji

* Update emoji

* Add changelog

* Update from PR feedback

* Revert "Add changelog"

This reverts commit 03656848eb.

* Revert "Minor change to trigger e2e tests"

This reverts commit 64e3a87b2d.

---------

Co-authored-by: Jon Lane <jon.lane@automattic.com>
2024-06-04 13:03:38 -07:00
Adrian Moldovan 9b251ce318
[testing workflows] Add test environments and deprecate the daily e2e workflow (#48101)
* Deprecate pr-smoke-test.yml

* Add configs for each environment

* Add scripts

* Add arguments to playwright command

* Update pnpm scripts and existing e2e projects

* Fix shellcheck warning

* Update test env start for Gutenberg stable run

* Add changelog

* Remove unused var

* Set the USE_WP_ENV var

* Configure daily e2e jobs

* Rename some steps
2024-06-03 14:12:45 -07:00
Adrian Moldovan 97e7f86a15
Add the slack-test-report util (#47805)
* Add the slack-test-report util

* Error if the required GitHub context variables are not set

* Error if the required SLACK_CHANNEL env var is not set

* Use a more generic message to be able to report on jobs that are not tests, like build

* Add the alert-on-failure job in ci

* Run for pull_request to test the job

* Setup monorepo first

* Force a linting error

* Better logging

* Use inputs.trigger as report name if set

* Force an e2e test failure

* Set the commit message in the notification

* Use INPUT_TRIGGER for all notifications

* Revert test changes
2024-05-24 13:18:54 -07:00
Adrian Moldovan 835625c329
Fix reporting step for daily e2e runs (#47764)
* Add condition on github.event_name for lint jobs

* Update a file to trigger a lint job

* Update condition to run only on pull_request

* Revert test change

* Always run the Slack notification step

* Add pull_request test trigger

* Update report step

* Remove branch filter

* Try with push event and branch filter

* Pass secrets to ci.yml

* Update testIgnore pattern for Gutenberg project

* Add changelog

* Remove the test push trigger

* One more test run

* Removed push test trigger
2024-05-23 09:12:10 -07:00
Adrian Moldovan f1489892aa
[ci] Only run lint jobs on pull_request event (#47652) 2024-05-23 07:16:07 -07:00
Adrian Moldovan 13ab9007f6
[testing workflows] Add workflow_call trigger in CI workflow - support daily and other types of runs (#47612)
* Add workflow_call event

* Add tests-daily-run.yml

* Remove workflow version

* Update input value

* Remove secrets from caller workflow

* Rename job

* Check context

* Check context

* Check workflow_call inputs

* Override github.event_name with inputs.trigger

* Fix input name

* Add config for daily e2e tests with Gutenberg

* Update env commands for Gutenberg run

* Add disableHpos env variable in ci-utils

* Revert the trigger to push

* Revert the env variable setting in ci-jobs

* Revert the env variable setting in ci-jobs

* Rebuild utils

* Update test env start command

* Define the Gutenberg Playwright project

* Use path relative to workspace root

* Update events for test projects

* Update events for test projects

* Add changelogs

* Remove the e2e tests with Gutenberg tests from daily runs

* Set the schedule and workflow_dispatch triggers

* Add --project default flag to only run the default tests in CI

* Renamed snapshots to match the new Playwright project name
2024-05-22 11:04:44 -07:00
Adrian Moldovan 0522511a72
[testing worklflows] Consolidate the Slack channel used for test reports (#47611) 2024-05-22 11:01:51 -03:00
Adrian Moldovan cf6e01d0d6
[ci-jobs] Add disableHpos test environment variable (#47619)
* Add disableHpos test env variable

* Update events for HPOS disabled tests

* Add pull_request event to test the env

* Remove the pull_request event

* Add changelog

* Make the disableHpos var a boolean

* Also set the env in ci.yml

* Remove console log statement

* Print woocommerce_custom_orders_table_enabled

* Run test setup script again to check output

* Capture env setup output and print it

* Get woocommerce_custom_orders_table_enabled value in test env script

* Get more details on HPOS setting

* Remove the DISABLE_HPOS logic from test env setup script

* Use DISABLE_HPOS instead of ENABLE_HPOS everywhere

* Turn off no-console and jest/no-standalone-expect eslint rules for api core tests

* Test API tests with HPOS disabled

* Fix eslint error

* Clean up debug statements

* Remove pull_request test trigger
2024-05-21 18:33:43 -03:00
Adrian Moldovan ccba59ce05
[ci] Only evaluate jobs statuses for pull request event (#47615)
* Condition evaluate status job by github event

* Test reverse condition

* Test with push event

* Revert condition to match pull_request
2024-05-21 18:28:24 -03:00
Adrian Moldovan 9b8063f6b2
[testing workflows] Add optional jobs in CI workflow (#47414)
* Include performance tests job in the needed jobs for evaluation

* Use Github api to get the workflow jobs status

* Evaluate project-jobs

* See results

* Check for cancelled jobs

* Add evaluate-project-jobs.js and rename optional jobs

* Fix script path

* Checkout first

* And then fix the path again.

* Update env variable name and be more detailed in what are the required variables

* Use a flat matrix env variable

* Update to trigger test jobs

* Use the jobs name to evaluate optional requirement

* Fix conditions

* Prettier print

* Test lint job failure

* Remove unused MATRIX env variable

* Fix test

* Remove unused MATRIX variable check

* Only run my account e2e tests

* Remove unused function

* Nicer console printing

* Revert change that triggers failing lint job

* Force failing e2e test

* Revert e2e test command and forced failure

* Added test data and test mode

* Added more test data

* Fixed lint errors and warnings

* Exclude .github folder from eslintignore

* Change to trigger everything

* Revert change to trigger everything
2024-05-16 13:31:17 -07:00
Jonathan Lane ad19ff51a3
Add Buildkite reporting for e2e PR and daily runs (#47558)
* Add Buildkite reporting for e2e PR runs

* Set up Buildkite reporting on daily runs

---------

Co-authored-by: Jon Lane <jon.lane@automattic.com>
2024-05-16 15:31:21 -03:00
Adrian Moldovan 7b96dfca67
[e2e tests] Remove BuildKite reporter (#47449)
* Remove BuildKite reporter

* Changelog

* Remove BuildKite reporter from api tests
2024-05-14 10:25:17 +01:00
Adrian Moldovan 981fca33d6
[testing workflows] Fix for multi-line commit message not triggering tests (#47353) 2024-05-10 10:12:36 -03:00
Jonathan Lane 396c29d3ec
Install and set up BuildKite test analytics (#47202)
Co-authored-by: Jon Lane <jon.lane@automattic.com>
Co-authored-by: Adrian Moldovan <3854374+adimoldovan@users.noreply.github.com>
2024-05-10 11:47:23 +03:00
Adrian Moldovan de91a0881d
[ci-jobs] Add support for github event and include HPOS disabled e2e tests in ci (#46922)
* Update env setup script to disable hpos

* Use DISABLE_HPOS instead of ENABLE_HPOS

* Log HPOS state

* Configure project for core e2e with HPOS disabled

* Add event argument for ci-jobs util

* Add event option for test jobs

* Add support for github events

* Add changelog

* Configure HPOS tests to run on pull_request

* Fix utils tests for undefined commandVars

* Added some tests for the event configuration

* Revert event for HPOS e2e to push

* Use matrix name in artifact name to avoid duplication

* Test with pull_request event

* Use job-index for unique artifacts names

* Revert event for HPOS e2e to push

* Add api tests for HPOS disabled

* Use unique artifact name for api tests

* Revert event for HPOS disabled api tests to push

* Rebuild monorepo utils to fix a merge conflict

* Updated wording
2024-05-01 10:36:24 -07:00
dependabot[bot] 3d8e20c62c
Bump octokit from 2.1.0 to 3.1.2 (#42891)
* Bump octokit from 2.1.0 to 3.1.2

Bumps [octokit](https://github.com/octokit/octokit.js) from 2.1.0 to 3.1.2.
- [Release notes](https://github.com/octokit/octokit.js/releases)
- [Commits](https://github.com/octokit/octokit.js/compare/v2.1.0...v3.1.2)

---
updated-dependencies:
- dependency-name: octokit
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Rename env variable

* Add changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jon Lane <jon.lane@automattic.com>
2024-04-26 10:52:25 -03:00
Jonathan Lane 9888ee7f4b
Updated deprecated gh-script action to node 20 variant (#46673)
* Updated deprecated gh-script action to node 20 variant

* Minor change in e2e test to trigger CI

* Revert e2e test change

---------

Co-authored-by: Jon Lane <jon.lane@automattic.com>
2024-04-17 12:31:16 +01:00
Adrian Moldovan 1fee0439d3
[testing workflows] Move metrics job to ci.yml (#46471)
* Move deprecated workflows to deprecated folder

* Configure new test job for metrics

* Add changelog

* Update start command

* Update start command

* Archive metrics results

* Fix script path

* Fix name in archive results condition

* Add the actual run command for pull_request event

* Run with pnpm --filter

* Test run for push event

* Revert test conditions

* Add WP_ARTIFACTS_PATH env variable

* Update path to readme.txt

* Revert github event conditions

* Add CODEVITALS_PROJECT_TOKEN env

* Set WP_ARTIFACTS_PATH on job level

* Add blocks templates in the changes list
2024-04-12 11:46:01 -03:00
Adrian Moldovan 9df545046a
[test reporting] Include list of failed tests in Slack notifications (#46360) 2024-04-10 10:10:08 +03:00
Adrian Moldovan 4e9fc5a90f
[testing workflows] Tweak conditions for Slack alerts and reports publishing jobs (#46031) 2024-04-02 14:40:24 -03:00
Adrian Moldovan 7d6d2c94dd
Include e2e, api and performance tests in ci.yml (#45190)
Co-authored-by: Christopher Allford <6451942+ObliviousHarmony@users.noreply.github.com>
Co-authored-by: Ron Rennick <ron@ronandandrea.com>
2024-03-27 17:21:49 +02:00
jonathansadowski a7f9139b33
Fix post-merge CI / update ci-jobs util to make base-ref optional (#44055)
* Update ci-jobs util to make base-ref optional

* Fix linting error

* Update baseRef option to be easier to read in ci.yml
2024-01-26 13:52:55 -06:00
Christopher Allford 31c170f80c
Added Missing Lint Build (#43665)
Unfortunately, it looks like we need to build in order
to be able to lint. This makes sense since some
projects will lint imports from others that haven't
been built yet.
2024-01-15 12:30:54 -08:00