Tests should be consistent. That is true for our unit tests suite, but it is something that is harded to achieve for functional tests. Our end to end tests often times fail due to factors outside of our control, and simply manually restarting the Travis build is enough to make them pass (example: https://github.com/woocommerce/woocommerce/pull/21150#issuecomment-415132390). This commits uses `travis_retry` to make Travis automatically retry a maximum of three times to run WC e2e tests in case of a failure.
This commit changes the travis codecoverage from using xdebug to phpdbg, phpdbg seems much faster and gives similar results.
Reason for switching is we have been running into constant timeouts on our codecoverage due to the 50min job limit on travis, which means our codecoverage has not been updated in a couple of months.
* Remove xdebug as it slows tests down, switch to using phpdbg for code coverage.
* Update parameters for phpdbg
* It is qrr not qqr
* Include vendor/bin path when using phpdbg
* Use PHP 7.1 to run phpdbg
* Update phpunit dire
* Include $HOME in phpdbg call to phpunit
* Set no memory limit to avoid out of memory errors.
* Assign timeout group to test_request_url test for paypal and do not execute that on coverage as it causes a memeory timeout. Test needs optimization to run for code coverage.
* @covers usage for methods should be prefixed with ::
PR #17680 added a new PHP 7.1 Travis build job to generate code coverage report. PHPCS was configured to run on all PHP 7.1 build jobs. So this means that after #17680 was merged, Travis started running PHPCS twice.
This commit fixes this issue by setting a new environment variable called `$RUN_PHPCS` and using this variable, instead of the PHP version, to decide when to run PHPCS.
This commit changes Travis configuration file to specify a list of branches that it should build. This way Travis will only create a new build when a commit is pushed to the master branch and release branches and when tags are created. The behavior for PRs is not modified and PR creation and updates will still trigger builds.
Doing this to speed up Travis build by preventing it from running two builds for PR that are created from a branch of the same repository. Before this change, Travis would create a build when the branch is pushed to GitHub (continuous-integration/travis-ci/push) and another one when the PR is created (continuous-integration/travis-ci/pr). For an example, see https://github.com/woocommerce/woocommerce/pull/17680.
It is not necessary to declare environment variables for a specific build job if they are the same as the environment variables declared for all build jobs.
Travis build is taking about 40 minutes to complete, and that is mostly because of the generation of the code coverage report.
To address that, this commit changes Travis configuration to run the command to generate code coverage report on a separate non-blocking Travis job. This way once the jobs that run the tests finishes, Travisi will mark the build as successful and will keep running code coverage on a separate job.
Rodrigo Primo
Gerhard Potgieter
Brent Sessions
Claudio Sanches
Mike Jolley
Akeda Bagus
Ramon van Belzen
Jon Surrell
Claudio Sanches
Aristeides Stathopoulos