diff --git a/End-to-end-Testing.md b/End-to-end-Testing.md index 32503e5..fe46683 100644 --- a/End-to-end-Testing.md +++ b/End-to-end-Testing.md @@ -1,220 +1 @@ - - -## Table of Contents - -- [Overview](#overview) -- [Getting started](#getting-started) -- [Troubleshooting broken tests](#troubleshooting-broken-tests) -- [Using the end-to-end components for testing your extensions](#using-the-end-to-end-components-for-testing-your-extensions) -- [Contributing](#contributing) - - - -## Overview - -End-to-end testing tests whether the whole flow of an application is working as expected. This is different from unit testing, which tests only individual components to verify they work properly. Unit testing doesn't test whether visible components are rendering and working correctly and doesn't test the system as a whole (e.g. "can a user complete the checkout process successfully?"). End-to-end testing can fill in those gaps. - -End-to-end testing will be an important part of ensuring WooCommerce is working correctly and changes don't break anything. It automates the manual process of going through all of the different customer-facing and admin-facing pages, performing actions on those pages, and verifying those actions yield the expected result. This increases the speed and accuracy of the QA process, and it increases confidence that nothing is broken before releases. The long-term plan for the project is to ideally be able to automatically go through [the whole core testing checklist](https://github.com/woocommerce/woocommerce/wiki/Core-Testing-Checklist). - -The end-to-end tests bundled in WooCommerce can be used to verify WooCommerce is working correctly on its own, or they can be used to verify WooCommerce is not broken after enabling an extension. The latter should be helpful for extension developers that want to verify they didn't break WooCommerce anywhere with their extension. If your extension substantially changes WooCommerce screens, you may have to use a custom test suite. Instructions for building a custom test suite using the WooCommerce end-to-end components are further down this page. - - -## Pre-requisites - -### Install NodeJS - -Install NodeJS on Mac: - -```bash -brew install node -``` - -### Install Docker - -Install Docker Desktop if you don't have it installed: - -- [Docker Desktop for Mac](https://docs.docker.com/docker-for-mac/install/) -- [Docker Desktop for Windows](https://docs.docker.com/docker-for-windows/install/) - -Once installed, you should see `Docker Desktop is running` message with the green light next to it indicating that everything is working as expected. - -Note, that if you install docker through other methods such as homebrew, for example, your steps to set it up will be different. The commands listed in steps below may also vary. - -## Configuration - -This section explains how e2e tests are working behind the scenes. These are not instructions on how to build environment for running e2e tests and run them. If you are looking for instructions on how to run e2e tests, jump to [Running tests](#running-tests). - -### Test Environment - -We recommend using Docker for running tests locally in order for the test environment to match the setup on Travis CI (where Docker is also used for running tests). [An official WordPress Docker image](https://github.com/docker-library/docs/blob/master/wordpress/README.md) is used to build the site. Once the site using the WP Docker image is built, the current WooCommerce dev branch is being copied to the `plugins` folder of that newly built test site. No WooCommerce Docker image is being built or needed. - -### Environment Variables - -During the process of Docker building a container with test site for running tests, site URL is being set. Admin and customer users are also being created in advance with details specified in the `docker-compose.yaml` file. As a result, there is `./tests/e2e-tests/config/default.json` file that contains pre-set variables needed to run the test: - -``` -{ - "url": "http://localhost:8084/", - "users": { - "admin": { - "username": "admin", - "password": "password" - }, - "customer": { - "username": "customer", - "password": "password" - } - } -} -``` - -If you changed either site URL or one of the users details in the `docker-compose.yaml` file, you'd need to copy the content of the `default.json`, paste it to `test:e2e.json` and edit it further there to match your own setup. - -### Jest test sequencer - -[Jest](https://jestjs.io/) is being used to run e2e tests. By default, jest runs tests ordered by the time it takes to run the test (the test that takes longer to run will be run first, the test that takes less time to run will run last). Jest sequencer introduces tools that can be used to specify the order in which the tests are being run. In our case, they are being run in alphabetical order of the directories where tests are located. This way, tests in the new directory `activate-and-setup` will run first. - -Setup Wizard e2e test (located in `activate-and-setup` directory) will run before all other tests. This will allow making sure that WooCommerce is activated on the site and for the setup wizard to be completed on a brand new install of WooCommerce. - -## Running tests - -### Prep work for running tests - -- `cd` to the WooCommerce plugin folder - -- `git checkout master` or checkout the branch where you need to run tests - -- Run `npm install` - -- Run `composer install` - -- Run `npm run build` - -- Run the following command to build the test site using Docker: `docker-compose up` and watch the site being built. Note that it may take a few minutes the first time you do that. The process is considered completed when the messages letting you know that WordPress was installed, WooCommerce was activated and users created will be displayed: - -``` -wordpress-cli_1 | Success: WordPress installed successfully. -wordpress-cli_1 | Plugin 'woocommerce' activated. -wordpress-cli_1 | Success: Activated 1 of 1 plugins. -wordpress-cli_1 | Success: Created user 2. -woocommerce_wordpress-cli_1 exited with code 0 -woocommerce_wordpress-cli_1 exited with code 0 -``` - -For more Docker commands, scroll down to [Docker basics](#docker-basics). - -- Open new terminal window and `cd` to the current branch again. - -- Run the following command to make sure the containers were built and running: `docker ps`. You should see the 2 following containers: - -- `woocommerce_wordpress-woocommerce-dev` -- `mariadb:10.4` - -- Navigate to `http://localhost:8084/`. If everything went well, you should be able to access the site. - -### How to run tests in headless mode - -To run e2e tests in headless mode use the following command: - -```bash -npm run test:e2e -``` - -### How to run tests in non-headless mode - -Tests are being run headless by default. However, sometimes it's useful to observe the browser while running tests. To do so, you can run tests in a non-headless (dev) mode: - -```bash -npm run test:e2e-dev -``` - -The dev mode also enables SlowMo mode. SlowMo slows down Puppeteer’s operations so we can better see what is happening in the browser. You can adjust the SlowMo value by editing `PUPPETEER_SLOWMO` variable in `./tests/bin/e2e-test-integration.js` file. The default `PUPPETEER_SLOWMO=50` means test actions will be slowed down by 50 milliseconds. - -### How to run an individual test - -To run an individual test, use the direct path to the spec. For example: - -```bash -npm run test:e2e ./tests/e2e-tests/specs/wp-admin/wp-admin-product-new.test.js -``` - -### How to skip tests - -To skip the tests, use `.only` in the relevant test entry to specify the tests that you do want to run. - -For example, in order to skip Setup Wizard tests, add `.only` to the login and activation tests as follows in the `setup-wizard.test.js`: - -``` -it.only( 'Can login', async () => {} -``` - -``` -it.only( 'Can make sure WooCommerce is activated. If not, activate it', async () => {} -``` - -As a result, when you run `setup-wizard.test.js`, only the login and activate tests will run. The rest will be skipped. You should see the following in the terminal: - -``` - PASS tests/e2e-tests/specs/activate-and-setup/setup-wizard.test.js (11.927s) - Store owner can login and make sure WooCommerce is activated - ✓ Can login (7189ms) - ✓ Can make sure WooCommerce is activated. If not, activate it (1187ms) - Store owner can go through store Setup Wizard - ○ skipped Can start Setup Wizard - ○ skipped Can fill out Store setup details - ○ skipped Can fill out Payment details - ○ skipped Can fill out Shipping details - ○ skipped Can fill out Recommended details - ○ skipped Can skip Activate Jetpack section - ○ skipped Can finish Setup Wizard - Ready! section - Store owner can finish initial store setup - ○ skipped Can enable tax rates and calculations - ○ skipped Can configure permalink settings -``` - -You can also use `.skip` in the same fashion. For example: - -``` -it.skip( 'Can start Setup Wizard', async () => {} -``` - -Finally, you can aply both `.only` and `.skip` to `describe` part of the test: - -``` -describe.skip( 'Store owner can go through store Setup Wizard', () => {} -``` - -## Writing tests - -We use the following tools to write e2e tests: - -- [Puppeteer](https://github.com/GoogleChrome/puppeteer) – a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol -- [jest-puppeteer](https://github.com/smooth-code/jest-puppeteer) – provides all required configuration to run tests using Puppeteer -- [expect-puppeteer](https://github.com/smooth-code/jest-puppeteer/tree/master/packages/expect-puppeteer) – assertion library for Puppeteer - -Tests are kept in `tests/e2e-tests/specs` folder. - -The following packages are being used to write tests: - -- `e2e-test-utils` - End-To-End (E2E) test utils for WordPress. You can find the full list of utils [here](https://github.com/WordPress/gutenberg/tree/master/packages/e2e-test-utils); -- `puppeteer-utils` - Utilities and configuration for running puppeteer against WordPress. See details in the [package's repository](https://github.com/Automattic/puppeteer-utils). - -## Debugging tests - -For Puppeteer debugging, follow [Google's documentation](https://developers.google.com/web/tools/puppeteer/debugging). - -## Docker basics - -### How to stop and restart Docker - -- Press `Ctrl+C` in the terminal window where the containers are running -- Stop the container(s) using the following command: `docker-compose down` -- Restart the containers using the following command: `docker-compose up` - -### How to stop Docker and do a clean restart - -Steps below will allow building a brand new site with a clean DB and no data as it was built initially: - -- Press `Ctrl+C` in the terminal window where the containers are running -- Stop the container(s) and delete all volumes using the following command: `docker-compose down -v` -- Restart the containers using the following command: `docker-compose up --build` \ No newline at end of file +Please refer to the [new e2e testing documentation](https://github.com/woocommerce/woocommerce/tree/master/tests/e2e). \ No newline at end of file