removing QIT for now
This commit is contained in:
Brent MacKinnon
parent
6dec084ab4
commit
453a6ab550
Binary file not shown.
|
Before Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 536 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 383 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 385 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 34 MiB |
|
|
@ -1,33 +0,0 @@
|
|||
# Authenticating
|
||||
|
||||
A vendor admin user is required to be able to view and execute tests using QIT.
|
||||
|
||||
## Dashboard
|
||||
|
||||
For the QIT dashboard:
|
||||
|
||||
- Log in to WooCommerce.com with your vendor account.
|
||||
- Click on `Vendor Dashboard` button to be taken to your vendor dashboard, which can be found on the My Account page once you've logged in:
|
||||
|
||||
|
||||
|
||||
- Don't see this button? You may not be the vendor admin on the account. Reach out to someone else in the organization (usually the person that handles uploading the extension for publishing) to see if they have access.
|
||||
|
||||
> Even if you don't have access in the UI, you'll still be able to leverage the CLI. Please see the section below for a guide on how to do this.
|
||||
|
||||
## CLI
|
||||
|
||||
In order to be able to use the CLI tool, you'll need to be a vendor admin that can create a [WordPress application password](https://make.wordpress.org/core/2020/11/05/application-passwords-integration-guide/) in order to authenticate. Follow the steps below to authenticate with the QIT CLI:
|
||||
|
||||
- [Download](https://github.com/woocommerce/qit-cli/releases/latest/) the latest version of QIT CLI and [Install it](qit-cli/getting-started#installing-qit)
|
||||
- Depending on how you've installed the QIT CLI, run `./vendor/bin/qit partner:add`
|
||||
- Follow the steps to generate an application password
|
||||
- Enter the application password and username in the CLI
|
||||
|
||||
## Giving access to other developers to use the QIT
|
||||
|
||||
Sometimes you want to give access to other developers in your organization to run tests using the QIT, but you might not want to give them access to the WooCommerce.com account that can manage the extension in the marketplace, as it gives developers access they don't need, such as managing your extensions in the marketplace, etc.
|
||||
|
||||
Luckily, you can share with them the QIT Application Password, as these application passwords are restricted to only run and view test runs. They are special application passwords with limited access that can only run and view tests using QIT.
|
||||
|
||||
> Our roadmap includes plans to add the ability to create and revoke QIT-specific access tokens to make this particular workflow and use-case more manageable.
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
# Contact Us
|
||||
|
||||
We're here to help! If you have any questions, comments, or concerns, please don't hesitate to reach out to us. You can contact us by sending an email to [qit@woocommerce.com](mailto:qit@woocommerce.com).
|
||||
|
||||
## Report Issues
|
||||
|
||||
If you encounter any issues with our QIT CLI tool, please let us know by opening an issue on our GitHub repository. You can access the repository here: [https://github.com/woocommerce/qit-cli/issues](https://github.com/woocommerce/qit-cli/issues)
|
||||
|
||||
## Provide Feedback
|
||||
|
||||
We value your feedback and suggestions. Please take a moment to let us know how we're doing. Your input will help us continue to improve.
|
||||
|
|
@ -1,98 +0,0 @@
|
|||
# GitHub Workflows Example
|
||||
|
||||
The QIT GitHub Workflows are examples of integrating the Quality Insights Toolkit into GitHub. You can delegate tests to the QIT and integrate it as part of your PR approval process, when a release is created, and more.
|
||||
|
||||
For more information on how [GitHub Actions](https://docs.github.com/en/actions) work, please see the official GitHub documentation.
|
||||
|
||||
The examples below can be tweaked based on your needs, and use a fictional `woocommerce-product-feeds` extension to run the tests against. There's a few [GitHub Secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) that need to be configured for this flow (feel free to rename these to whatever makes the most sense for you and your team):
|
||||
|
||||
- `PARTNER_USER`: Your WooCommerce.com username.
|
||||
- `PARTNER_SECRET`: Your [WordPress Application Password](https://make.wordpress.org/core/2020/11/05/application-passwords-integration-guide/).
|
||||
|
||||
## Activation test example
|
||||
|
||||
```yaml
|
||||
name: QIT Activation Test
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
permissions:
|
||||
pull-requests: write
|
||||
jobs:
|
||||
qit_activation:
|
||||
name: QIT Activation
|
||||
runs-on: ubuntu-20.04
|
||||
env:
|
||||
NO_COLOR: 1
|
||||
QIT_DISABLE_ONBOARDING: yes
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v3
|
||||
- name: Build your plugin zip (Example)
|
||||
run: zip -r my-extension.zip my-extension
|
||||
- name: Install QIT via composer
|
||||
run: composer require woocommerce/qit-cli
|
||||
- name: Add Partner
|
||||
run: ./vendor/bin/qit partner:add --user='${{ secrets.PARTNER_USER }}' --application_password='${{ secrets.PARTNER_SECRET }}'
|
||||
- name: Run Activation Test
|
||||
id: run-activation-test
|
||||
run: ./vendor/bin/qit run:activation my-extension --zip=my-extension.zip --wait > result.txt
|
||||
- uses: marocchino/sticky-pull-request-comment@v2
|
||||
if: failure()
|
||||
with:
|
||||
header: QIT Activation Result
|
||||
recreate: true
|
||||
path: result.txt
|
||||
```
|
||||
|
||||
## Security test example
|
||||
|
||||
```yaml
|
||||
name: QIT Security Test
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
permissions:
|
||||
pull-requests: write
|
||||
jobs:
|
||||
qit_security:
|
||||
name: QIT Security Test
|
||||
runs-on: ubuntu-20.04
|
||||
env:
|
||||
NO_COLOR: 1
|
||||
QIT_DISABLE_ONBOARDING: yes
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v3
|
||||
- name: Build your plugin zip (Example)
|
||||
run: zip -r my-extension.zip my-extension
|
||||
- name: Install QIT via composer
|
||||
run: composer require woocommerce/qit-cli
|
||||
- name: Add Partner
|
||||
run: ./vendor/bin/qit partner:add --user='${{ secrets.PARTNER_USER }}' --application_password='${{ secrets.PARTNER_SECRET }}'
|
||||
- name: Run Activation Test
|
||||
id: run-activation-test
|
||||
run: ./vendor/bin/qit run:security my-extension --zip=my-extension.zip --wait > result.txt
|
||||
- uses: marocchino/sticky-pull-request-comment@v2
|
||||
if: failure()
|
||||
with:
|
||||
header: QIT Security Result
|
||||
recreate: true
|
||||
path: result.txt
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
### Required permissions
|
||||
|
||||
Depending on how your repository is set up, you may need to adjust the permissions step to the following:
|
||||
|
||||
```yaml
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
```
|
||||
|
||||
### Verify the zip is built correctly
|
||||
|
||||
If you see any failures in the pipeline, such as an activation test that fails without a result, it could be due to the way that the GitHub action is building the zip file. Double check that the zip file created by the workflow doesn't add in an extra folder to the file structure. To verify that the zip is valid to be used by QIT, you can test uploading it in the WordPress admin screen under `Plugins > Add new` and test uploading and activating the zip file. If it works, then it will work with QIT.
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
# Introduction
|
||||
|
||||
The Quality Insights Toolkit (QIT) is an initiative by WooCommerce designed to provide extension developers in the [Woo Marketplace](https://woocommerce.com/products/) with managed automated tests.
|
||||
|
||||
To ensure that all extensions in the Woo Marketplace meet our quality standards, we run a series of automated tests. As part of our commitment to supporting developers, we also provide ways for developers to easily integrate these tests into their development workflows.
|
||||
|
||||
## Quick Start Guide
|
||||
|
||||
1. `composer install woocommerce/qit-cli`
|
||||
2. `./vendor/bin/qit` to authenticate with your WooCommerce.com developer account.
|
||||
|
||||
## Benefits for You as a Developer
|
||||
|
||||
- You can enjoy out-of-the-box tests for all the extensions you sell on the Marketplace
|
||||
- Your extensions will undergo continuous testing, including against new releases of PHP, WooCommerce, and WordPress
|
||||
- You can have increased peace of mind knowing that you'll be alerted if the tests catch anything with a future release or an edge case
|
||||
- You can take advantage of integration with GitHub Workflows
|
||||
- You can easily integrate the testing tool with your development workflow using the terminal tool
|
||||
- You can run some tests with other extensions active at the same time, including those that may be known to cause issues when activated alongside yours.
|
||||
|
||||
## Benefits for Users of the WooCommerce Ecosystem
|
||||
|
||||
- Users can enjoy increased reliability when updating WooCommerce extensions on the Marketplace
|
||||
- Users can have confidence to update PHP, WordPress, or WooCommerce versions without fear of breaking their extensions
|
||||
- There will be less potential security issues for users of the WooCommerce Ecosystem
|
||||
- Users can have more assurance that all extensions will continue to work as expected when activated alongside each other, ensuring a seamless experience.
|
||||
|
||||
## Who is this toolkit for?
|
||||
|
||||
This toolkit is for extension developers who are selling their extensions on the WooCommerce Marketplace.
|
||||
|
||||
## What types of tests are available?
|
||||
|
||||
- End-to-end
|
||||
- API
|
||||
- Activation
|
||||
- Security
|
||||
- PHPStan
|
||||
|
||||
## How can I use the toolkit?
|
||||
|
||||
Tests are executed automatically by us whenever you publish a new version of your extension on the WooCommerce Marketplace. You can also run tests manually using the following tools:
|
||||
|
||||
- [Dashboard](qit-dashboard/getting-started.md): A UI-based test runner and test results viewer, available in your WooCommerce dashboard.
|
||||
- [CLI](qit-cli/getting-started.md): A CLI tool that allows you to run and view tests, including against development builds.
|
||||
- [GitHub Workflows](github-workflows.md): GitHub workflow files that allow running tests regularly with QIT as part of a GitHub development workflow.
|
||||
|
||||
## Why does this toolkit exist?
|
||||
|
||||
The primary goal of these tools is to assist extension developers to easily integrate a variety of tests into their development workflows, and promote and encourage quality around WooCommerce extensions available in the marketplace.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 770 KiB |
|
|
@ -1,52 +0,0 @@
|
|||
# Getting started
|
||||
|
||||
The QIT CLI is a command line interface tool that allows you to run automated tests in the cloud against extensions available in the WooCommerce Marketplace, powered by the Quality Insights Toolkit test runner.
|
||||
|
||||
|
||||
|
||||
## Requirements
|
||||
|
||||
- PHP 7.2.5 or higher
|
||||
- Unix environment (Linux, macOS, Windows WSL)
|
||||
- Composer
|
||||
|
||||
## Installing QIT
|
||||
|
||||
You can install QIT in three different ways:
|
||||
|
||||
### Per Project
|
||||
|
||||
1. Run `composer require woocommerce/qit-cli --dev`
|
||||
2. Execute `./vendor/bin/qit` to authenticate with your WooCommerce.com Partner Developer account.
|
||||
|
||||
### Globally Using Composer
|
||||
|
||||
1. Run `composer global require woocommerce/qit-cli`
|
||||
2. Execute `qit` to authenticate with your WooCommerce.com Partner Developer account. Ensure that the Composer bin folder is in your PATH. [Example](https://stackoverflow.com/a/64545124).
|
||||
|
||||
### Globally Using `wget`
|
||||
|
||||
(Pro Tip: Opting for the Composer installation method simplifies the process of updating QIT in the future 😉)
|
||||
|
||||
1. Run `wget https://github.com/woocommerce/qit-cli/raw/trunk/qit`
|
||||
2. Execute `chmod +x qit`
|
||||
3. Move the file to a directory in your PATH, such as `sudo mv qit /usr/local/bin/qit`
|
||||
4. Run `qit` to authenticate with your WooCommerce.com Partner Developer account.
|
||||
|
||||
## Updating QIT
|
||||
|
||||
Updating QIT will depend on how you installed it.
|
||||
|
||||
### If you installed it per-project
|
||||
|
||||
1. Run `composer update woocommerce/qit-cli`
|
||||
|
||||
### If you installed it globally using Composer
|
||||
|
||||
1. Go to your global composer directory `cd $(composer global config bin-dir --absolute)`
|
||||
2. Run `composer update woocommerce/qit-cli`
|
||||
|
||||
#### If you installed it with `wget`
|
||||
|
||||
1. Delete the old binary `which qit` then `rm` it.
|
||||
2. Repeat the installation steps.
|
||||
|
|
@ -1,78 +0,0 @@
|
|||
# Running tests
|
||||
|
||||
The QIT CLI allows running tests against both extensions that are published and available for sale in the WooCommerce Extension Store as well as against development builds of an extension.
|
||||
|
||||
## Choosing the type of test to run
|
||||
|
||||
The commands to run tests are formatted as `run:<test-type>`. The CLI supports all of the current [test types](test-types.md) using the following commands:
|
||||
|
||||
- End-to-end
|
||||
- API
|
||||
- Activation
|
||||
- Security
|
||||
- PHPStan
|
||||
|
||||
For example, to run end-to-end tests, you'd run the following command: `./vendor/bin/qit run:e2e my-plugin-slug`.
|
||||
|
||||
## Testing a published extension
|
||||
|
||||
For running any kind of tests, you'll need the slug for the given extension you want to run the tests against. For example, to run end-to-end tests against an extension with the slug `my-extension`, you'd run the following command:
|
||||
|
||||
```shell
|
||||
qit run:e2e my-extension
|
||||
```
|
||||
|
||||
This will run the [WooCommerce Core E2E Test Suite](https://github.com/woocommerce/woocommerce/tree/trunk/plugins/woocommerce/tests/e2e-pw) against the WooCommerce extension with slug `my-extension` using the latest stable versions of WordPress and WooCommerce.
|
||||
|
||||
Since the tests are executed in the cloud, you can even close the terminal if you wish. You can see the result of this test after a while running `qit list-tests`, or `qit get 123`, where `123` is the test run ID. When the test finishes, the status will be updated to `Success`, `Warning`, or `Failed`. For more details on these commands, please see [Useful Commands](cli/useful-commands.md).
|
||||
|
||||
## Testing development builds
|
||||
|
||||
The QIT CLI supports testing development builds of extensions, so you can run any of the [test types](../test-types.md) against an unpublished version of your extension in the same QIT environment before publishing it to the WooCommerce Store.
|
||||
|
||||
!> Make sure the zipped version is a valid plugin. As this is installed on a test WordPress site, an invalid plugin will fail to install and cause the tests to fail.
|
||||
|
||||
Once you have a zipped up version of your extension you'd like to test with, use the `--zip` argument to pass in a path to the zip file containing your extension.
|
||||
|
||||
For example, to run end-to-end tests against your local build, you'd run the following command:
|
||||
|
||||
```shell
|
||||
qit run:e2e my-extension --zip=my-extension.zip
|
||||
```
|
||||
|
||||
## Seeing test runs and their results
|
||||
|
||||
Since the tests are executed in the cloud, you can even close the terminal. You can see the result of this test after a while by running one of the two commands below:
|
||||
|
||||
- Run `qit list-tests` to see a list of test runs.
|
||||
- Run `qit get 123` to get more details about a specific test run, where `123` is the test run ID.
|
||||
- Run `qit open 123` to open the report for test run `123` in the browser.
|
||||
|
||||
When the test finishes, the status will be updated to `Success`, `Warning`, or `Failed`. For more details on what these commands show, please see [Useful Commands](useful-commands.md).
|
||||
|
||||
## Specifying WooCommerce and WordPress versions
|
||||
|
||||
The QIT CLI supports testing against different versions of WooCommerce and WordPress. There are two arguments available to use, depending on your needs:
|
||||
|
||||
- `--woocommerce_version`
|
||||
- `--wordpress_version`
|
||||
|
||||
For example, to run activation tests against the RC 2 version of WooCommerce 7.0.0 and WordPress 6.0.1, you can run the following command:
|
||||
|
||||
`qit run:activation my-extension --woocommerce_version=7.0.0-rc.2 --wordpress_version=6.0.1`
|
||||
|
||||
?> If either these arguments are not supplied, then the tests will just run against the current versions of WooCommerce and WordPress.
|
||||
|
||||
For example, if you wanted to run against the latest WordPress but use a different version of WooCommerce, you can do so:
|
||||
|
||||
`qit run:e2e my-extension --woocommerce_version=6.0.0`
|
||||
|
||||
## Using optional features
|
||||
|
||||
You can also enable option features in your test environment by using the `--optional_features` flag. For example, to run an activation test with the High Performance Order Storage feature enabled, you can run the following command:
|
||||
|
||||
`qit run:activation my-extension --optional_features=hpos`
|
||||
|
||||
Currently, QIT supports enabling the following optional features:
|
||||
|
||||
- `hpos`: [High Performance Order Storage (HPOS)](https://developer.woocommerce.com/roadmap/high-performance-order-storage/)
|
||||
|
|
@ -1,127 +0,0 @@
|
|||
# Scripting
|
||||
|
||||
QIT CLI allows you to create robust scripts that can optimize your development workflow. Here is an example of a bash script used for authentication and running tests against a development build.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
For this example, we will assume the following directory structure. This can be in the same folder where you develop your plugin or in its own directory:
|
||||
|
||||
- .env
|
||||
- bin/qit.sh
|
||||
- vendor/bin/qit
|
||||
- build/extension.zip _(Assuming this is created by `npm run build`)_
|
||||
|
||||
## Environment Variables (.env)
|
||||
|
||||
Create a `.env` file in the root directory of your project and add your QIT user and application password:
|
||||
|
||||
```bash
|
||||
QIT_USER=foo
|
||||
QIT_APP_PASS=bar
|
||||
```
|
||||
|
||||
## Bash Script (bin/qit.sh)
|
||||
|
||||
This script authenticates the QIT_USER and then runs security tests against the extension build. If the 'partner:remove' command is not available, it adds a partner using `QIT_USER` and `QIT_APP_PASSWORD`. For more information on how authentication works with QIT, see our [documentation around authentication](https://woocommerce.github.io/qit-documentation/#/authenticating).
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -x # Verbose mode.
|
||||
|
||||
# Check if QIT_USER and QIT_APP_PASSWORD are set and not empty
|
||||
if [[ -z "${QIT_USER}" ]] || [[ -z "${QIT_APP_PASSWORD}" ]]; then
|
||||
echo "QIT_USER or QIT_APP_PASSWORD environment variables are not set or empty. Please set them before running the script."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# When QIT is run for the first time, it will prompt for onboarding. This will disable that prompt.
|
||||
export QIT_DISABLE_ONBOARDING=yes
|
||||
|
||||
# If QIT_BINARY is not set, default to ./vendor/bin/qit
|
||||
QIT_BINARY=${QIT_BINARY:-./vendor/bin/qit}
|
||||
|
||||
# Check if 'partner:remove' command is in the list of available commands
|
||||
if ! $QIT_BINARY list | grep -q 'partner:remove'; then
|
||||
echo "Adding partner with QIT_USER and QIT_APP_PASSWORD..."
|
||||
$QIT_BINARY partner:add --user="${QIT_USER}" --application_password="${QIT_APP_PASSWORD}"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to add partner. Exiting with status 1."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Run the security command
|
||||
echo "Running security command..."
|
||||
$QIT_BINARY run:security my-extension --zip=./../build/extension.zip --wait
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Failed to run security command. Exiting with status 1."
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
## Script Runner (Choose between NPM, Composer, Make)
|
||||
|
||||
Script runners can be used to execute our bash script `qit.sh`. You can choose the script runner that best suits your needs. Below you can find some examples we've put together for NPM, Composer, and Make.
|
||||
|
||||
The **build** command in this script is just an example, and should be modified to fit your actual build process that generates the plugin zip that can be installed in a WordPress site.
|
||||
|
||||
### NPM
|
||||
|
||||
Usage: `npm run qit-security`
|
||||
|
||||
<details>
|
||||
<summary>package.json</summary>
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Project",
|
||||
"version": "1.0.0",
|
||||
"scripts": {
|
||||
"qit-security": "npm run build && dotenv -e .env -- bash ./bin/qit.sh",
|
||||
"build": "zip -r build/extension.zip my-extension"
|
||||
},
|
||||
"devDependencies": {
|
||||
"dotenv-cli": "^7.2.1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Composer
|
||||
|
||||
Usage: `composer run qit-security`
|
||||
|
||||
<details>
|
||||
<summary>composer.json</summary>
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"build": "zip -r build/extension.zip my-extension",
|
||||
"qit-security": "export $(cat .env | xargs) && composer run-script build && ./bin/qit.sh"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Makefile
|
||||
|
||||
Usage: `make qit-security`
|
||||
|
||||
<details>
|
||||
<summary>Makefile</summary>
|
||||
|
||||
```shell
|
||||
include ./.env
|
||||
export
|
||||
|
||||
build:
|
||||
zip -r build/extension.zip my-extension
|
||||
|
||||
qit-security: build
|
||||
bash ./bin/qit.sh
|
||||
```
|
||||
|
||||
</details>
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
# Useful commands
|
||||
|
||||
To see the commands that the QIT CLI provides, you can simply run `./qit` to see the full list.
|
||||
|
||||
Some helpful commands to get started include:
|
||||
|
||||
- `extensions` - Lists the WooExtensions you have access to test. The list includes the ID of the extension and the slug:
|
||||
|
||||
```shell
|
||||
+----------------+-------------------------------------------------------------------------------------+
|
||||
| ID | Slug |
|
||||
+----------------+-------------------------------------------------------------------------------------+
|
||||
| 123 | my-extension |
|
||||
+----------------+-------------------------------------------------------------------------------------+
|
||||
```
|
||||
|
||||
- `list-tests` - Lists the test runs, including details around the results, the versions tested and the test type:
|
||||
|
||||
?> `Zip` for the version denotes that the test was ran against a [development version](running-tests.md#testing-development-builds) of the plugin.
|
||||
|
||||
```shell
|
||||
+--------+------------+-------+------------+---------+-----------+-------------------------------+
|
||||
| Run Id | Test | WP | WC | Status | Report | Name/Version |
|
||||
+--------+------------+-------+------------+---------+-----------+-------------------------------+
|
||||
| 344745 | security | 6.1.1 | 7.2.2 | warning | | My Extension (Zip) |
|
||||
| 344759 | e2e | 6.1.1 | 7.2.0-rc.2 | failed | Available | My Extension (1.0.0) |
|
||||
+--------+------------+-------+------------+---------+-----------+-------------------------------+
|
||||
```
|
||||
|
||||
- `get <run ID>` - Get a single test run using the run ID from the `list tests` command:
|
||||
|
||||
```bash
|
||||
Run Id 344745
|
||||
Test Type security
|
||||
Wordpress Version 6.1.1
|
||||
Woocommerce Version 7.2.2
|
||||
Status warning
|
||||
Is Development Yes
|
||||
Woo Extension My Extension
|
||||
```
|
||||
|
||||
?> If a report is available, you can go into the [QIT Dashboard to view the report](../qit-dashboard/viewing-test-results.md#viewing-test-logs) or view the link by running `get` and the test run ID:
|
||||
|
||||
```shell
|
||||
Run Id 361745
|
||||
Test Type e2e
|
||||
Wordpress Version 6.1.1
|
||||
Woocommerce Version 7.2.2
|
||||
Status failed
|
||||
Result Url https://testreport.url
|
||||
Woo Extension My Extension
|
||||
```
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 176 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 48 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 365 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 69 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 53 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 59 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 68 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 38 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 201 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 190 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 71 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 117 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 331 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.0 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 201 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 22 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 60 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 112 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 7.2 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 33 KiB |
|
|
@ -1,43 +0,0 @@
|
|||
# Getting started
|
||||
|
||||
The QIT Dashboard is a tool available to extension developers in the WooCommerce administrative interface. This tool allows developers to run a variety of tests with their extension installed in a test environment, configured with the latest version of WordPress and either the release candidate or latest version of WooCommerce. If you have multiple extensions, you can select the extension you’d like to run the tests against.
|
||||
|
||||
To get started, you'll need to log in to your account on WooCommerce.com. Once you log in to your account, click on "Vendor Dashboard" to go to your dashboard:
|
||||
|
||||
|
||||
|
||||
In the menu on the left-hand side, you'll see a new `Quality Insights` menu item:
|
||||
|
||||
|
||||
|
||||
Hovering over the menu will give you the three sections that make up the QIT Dashboard:
|
||||
|
||||
- All Tests
|
||||
- Create a Test
|
||||
- Notification Settings
|
||||
|
||||
More details on each of these options is provided below.
|
||||
|
||||
## All tests
|
||||
|
||||
|
||||
|
||||
The All Test page gives you insight into the extension the tests were ran against, the version of the extension, and what test type was ran. In addition, you can see what the results of the test runs were (such as failed, success, or pending if they’re currently running) and, in the case of end-to-end tests, view the Allure test report if the tests fail.
|
||||
|
||||
For more insight into the test results, please see [Viewing test results](viewing-test-results.md).
|
||||
|
||||
## Run a test
|
||||
|
||||
|
||||
|
||||
On this page, you can select [the type of test](../test-types) you'd like to run, which extension to test, and choose the versions of WooCommerce and WordPress to run the tests against.
|
||||
|
||||
For more details around creating a test, please see [Run a test](run-a-test.md).
|
||||
|
||||
## Notification settings
|
||||
|
||||
|
||||
|
||||
If any of the tests result in a failed test result, we will send out an email that contains details on the test type that failed and the versions of the extension, WooCommerce, and WordPress that the test failed for. Email notifications can be enabled or disabled for each test type. By default, emails will be sent to your account email address.
|
||||
|
||||
For more information around configuring notifications, see [Configure notifications](notifications.md).
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
# Configure notifications
|
||||
|
||||
To update the way notifications are sent for failed tests, navigate to the `Notification Settings` page under the `Quality Insights` menu:
|
||||
|
||||
|
||||
|
||||
After making changes, click the `Update Settings` button at the bottom of the page:
|
||||
|
||||
|
||||
|
||||
## Notifications by product
|
||||
|
||||
Notifications can be enabled and disabled both on a per-product basis as well as for all products that you sell in the store. To update these settings, select the specific product or select `All Products` to update notification settings for all of your products at once:
|
||||
|
||||
|
||||
|
||||
## Notifications by test type
|
||||
|
||||
You can enable and disable notifications for each [test type](../test-types.md) that is available by clicking on the toggle:
|
||||
|
||||
|
||||
|
||||
## Customize email recipients
|
||||
|
||||
By default we'll send test result emails to your account email address. If you'd like to override and send these emails to other recipients, please add an optional comma-separated list of email addresses to send the tests results to.
|
||||
|
||||
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
# Run a test
|
||||
|
||||
To start a new test, navigate to the `Run a Test` page under the `Quality Insights` menu:
|
||||
|
||||
|
||||
|
||||
On this page, you can configure the type of test to run, the product to run the tests against, and which release version:
|
||||
|
||||
|
||||
|
||||
Once you've selected the type of test you'd like to run, the product to run it against, and the release version, click the `Confirm and run test` button at the bottom to queue up the test to be ran:
|
||||
|
||||
|
||||
|
||||
Once the test has been queued up, you'll be redirected to the All Tests page and will see a success notification that the test has been created:
|
||||
|
||||
|
||||
|
||||
!> Note that it may take a moment before the test shows in the table. You can refresh the page or wait for the table to automatically refresh.
|
||||
|
||||
The test will then be displayed in the table, showing the version of the extension, the selected test type, the selected WooCommerce Release version with a `Pending` status:
|
||||
|
||||
|
||||
|
||||
Once the test has been picked up and the tests are running, the status will change to `Running`:
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
More details on each the options available on the `Create a Test` page can be found below:
|
||||
|
||||
## Test type
|
||||
|
||||
Select the [type of test](test-types.md) you'd like to run. The current options are available:
|
||||
|
||||
- End-to-end
|
||||
- API
|
||||
- Activation
|
||||
- Security
|
||||
- PHPStan
|
||||
|
||||
## Product
|
||||
|
||||
Select the product you'd like to run the tests against. All products that you have submitted and are available on the WooCommerce Store will be listed in the dropdown.
|
||||
|
||||
## Release
|
||||
|
||||
Select the release versions of WooCommerce and WordPress you'd like to execute the tests against. Currently there are two options available:
|
||||
|
||||
- The most recent releases of WordPress and WooCommerce.
|
||||
- The most recent release of WordPress and the latest release candidate version of WooCommerce.
|
||||
|
|
@ -1,65 +0,0 @@
|
|||
# Viewing test results
|
||||
|
||||
To view the tests that were created and their results, navigate to the `All Tests` page under the `Quality Insights` menu:
|
||||
|
||||
|
||||
|
||||
In the table on this page, all of the tests that you created will be shown in the list, starting with the most recent:
|
||||
|
||||
|
||||
|
||||
## Filtering results
|
||||
|
||||
You can filter the test results shown in the table by selecting one of the three available filters at the top of the page:
|
||||
|
||||
|
||||
|
||||
- Product: Filter by the product the tests were ran against.
|
||||
- WooCommerce Release: Filter by the WooCommerce release version the tests were ran against.
|
||||
- Test Types: Filter by the test type that was ran, such as e2e or activation.
|
||||
|
||||
You can also search the tests by providing keywords such as the product, the version, the test type, status or release:
|
||||
|
||||
|
||||
|
||||
## Viewing test logs
|
||||
|
||||
After a test run completes, you'll be able to view a variety of reports depending on the test type, and you can also share the link to the test results:
|
||||
|
||||
>  Click this icon to view the test results report.
|
||||
|
||||
>  Click this icon to copy a link to the test results to your clipboard to share with others.
|
||||
|
||||
### End-to-end tests
|
||||
|
||||
#### Successful runs
|
||||
|
||||
When an end-to-end test passes, you'll be able to view the report and see the tests that were ran by clicking on the view icon:
|
||||
|
||||
|
||||
|
||||
This will open a modal that you can scroll through and see the results of each test that was ran:
|
||||
|
||||
|
||||
|
||||
#### Failed test runs
|
||||
|
||||
If an end-to-end test run fails, an Allure test report will be generated. Click on the view icon to see the full details of the Allure test report:
|
||||
|
||||
|
||||
|
||||
### Security, Activation, and PHPStan tests
|
||||
|
||||
To view the test logs for these test types, click on view icon in the table for the extension test results you'd like to view:
|
||||
|
||||
|
||||
|
||||
This will open a modal where you can view the test results. For example, a failed Security test would show the following in the modal:
|
||||
|
||||
|
||||
|
||||
You'll also be able to view a log and share the result for successful tests as well:
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
# Quality Insights Toolkit
|
||||
|
||||
> ⚠️ **Notice:** This documentation is currently a **work in progress**. While it's open to the public for transparency and collaboration, please be aware that some sections might be incomplete or subject to change. We appreciate your patience and welcome any contributions!
|
||||
|
||||
The Quality Insights Toolkit (QIT) is an initiative by WooCommerce designed to provide current and prospective extension developers in the Woo Marketplace with managed automated tests.
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
# Test Options
|
||||
|
||||
The table below shows what options are available for each test type. For example, for Activation tests you can supply a WordPress version but for the security tests it isn't supported.
|
||||
|
||||
| | Activation | E2E | API | Security | PHPStan |
|
||||
| ---------------------------- | ---------- | --- | --- | -------- | ------- |
|
||||
| WordPress Versions | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| WooCommerce Versions | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| WooCommerce Features | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| PHP Version | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| Additional Extensions | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| Additional WordPress Plugins | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
|
||||
## WordPress and WooCommerce versions
|
||||
|
||||
You can specify the last 5 stable releases along with the most recent beta/release candidate to be used in the test environment.
|
||||
|
||||
?> Please note that for API and end-to-end tests, the WooCommerce version is currently restricted to specific versions as we work to make the tests backwards and forwards compatible. For the most up to date versions that we currently support, you can select the test type in the Viewer under `Run a Test` or, using the QIT CLI, check the help for each test type: `./qit run:api --help` and `./qit run:e2e --help`.
|
||||
|
||||
## WooCommerce features
|
||||
|
||||
QIT currently supports the following option WooCommerce features:
|
||||
|
||||
- [High Performance Order Storage (HPOS)](https://developer.woocommerce.com/roadmap/high-performance-order-storage/)
|
||||
|
||||
For more information on using optional features in your tests, check out [Testing WooCommerce Optional Features](cli/running-tests?id=using-optional-features).
|
||||
|
||||
## PHP versions
|
||||
|
||||
You can specify a PHP version from 7.4 to 8.2 to be used in the test environment.
|
||||
|
|
@ -1,243 +0,0 @@
|
|||
# Test Types
|
||||
|
||||
QIT supports the following test types:
|
||||
|
||||
- [End-to-end](#end-to-end-tests)
|
||||
- [API](#api-tests)
|
||||
- [Activation](#activation-tests)
|
||||
- [Security](#security-tests-beta)
|
||||
- [PHPStan](#phpstan-tests)
|
||||
|
||||
## End-to-End Tests
|
||||
|
||||
The end-to-end (e2e) test creates a temporary WordPress installation with WooCommerce and the extension under test installed, and uses a browser that is scripted to perform certain automated tasks, such as completing the WooCommerce onboarding wizard, creating a product, making a purchase as a customer, verifying the order details as an admin, tweaking tax settings, etc.
|
||||
|
||||
Then, it runs the [WooCommerce Core end-to-end tests](https://github.com/woocommerce/woocommerce/tree/trunk/plugins/woocommerce/tests/e2e-pw) against a store with your extension activated. These tests cover the [WooCommerce Core Critical Flows](https://github.com/woocommerce/woocommerce/wiki/Critical-Flows) to verify that a given extension does not break the default WooCommerce behaviors. Once the tests complete, the dashboard will show a Success or Failure test result. In the case of a failed test, a link to an Allure test report will be provided that allows you to dig into the details and see what failed and why.
|
||||
|
||||
> Currently, QIT can only run the WooCommerce Core E2E test suite. Future support for running your own E2E tests is planned.
|
||||
|
||||
### Example
|
||||
|
||||
This GIF is an example of the end-to-end test running. It performs a series of automated actions in a browser, such as creating a product, making a purchase, and verifying the order details as an admin, as fast as possible. The test is run against a store with the extension under test activated.
|
||||
|
||||
<details>
|
||||
<summary>Click to view GIF</summary>
|
||||
<img src="_media/e2e.gif"/>
|
||||
</details>
|
||||
|
||||
### What to do if it fails
|
||||
|
||||
If your end-to-end test is failing, please take the following steps:
|
||||
|
||||
- Check the Allure test report to see what failed and why. If you're unable to reproduce the issue manually, try re-running the test to see if it passes.
|
||||
- If the test continues to fail, it can be either because of a bug that should be fixed, or because your extension modifies the default WooCommerce behavior in a way that is unexpected by the automated tests.
|
||||
- We expect a certain amount of extensions to fail the end-to-end tests because they modify WooCommerce behaviors in ways that the tests are not designed to account for, such as modifying HTML selectors, etc. If you believe that is the case with your extension, please email us at qit@woocommerce.com and we can help you determine the best way to proceed, by either adapting the tests, suggesting some tweaks to your plugin, or by ignoring some tests specifically for your plugin.
|
||||
|
||||
### Understanding Allure Reports
|
||||
|
||||
For end-to-end test failures, an Allure test report will be generated and is available on the [All Test page](../dashboard/viewing-test-results.md). Allure reports provide a lot of great information to help troubleshoot and diagnose any test failures. For failures, screenshots and a stacktrace is provided. This section provides an overview of a report and where to go to view results. For a more detailed overview, see the official Allure documentation under [Report Structure](https://docs.qameta.io/allure-report/#_report_structure).
|
||||
|
||||
### Viewing a report
|
||||
|
||||
An Allure report is generated for end-to-end test failures and can be viewed by clicking the `View Report` button on the `Quality Insights > All Tests` page in the QIT Dashboard:
|
||||
|
||||
Following this link will open the Allure test results in a modal:
|
||||
|
||||
|
||||
|
||||
The `Suites` section shows what browser the tests were ran on, and the results of the tests (currently the tests are only ran in Chrome):
|
||||
|
||||
|
||||
|
||||
This page will provide a quick view into what percentage of tests passed, failed, or were skipped. Any failures will be reported in the `Categories` section under `Product defects`:
|
||||
|
||||
|
||||
|
||||
### Digging into the details
|
||||
|
||||
#### Successful results
|
||||
|
||||
For successful tests, you can view the details by going into the `Suites` section, either from the `Overview` page or by clicking the `Suites` menu option on the left-hand side. There, you can click into individual tests that passed and view the steps that were taken, the HTML selectors that were used, and what assertions were made:
|
||||
|
||||
|
||||
|
||||
#### Failed cases
|
||||
|
||||
When tests fail, Allure will provide the stack trace, the error that it encountered, and a screenshot of where it failed:
|
||||
|
||||
|
||||
|
||||
This allows you to dig through the steps that were taken by the test, download the stack trace, and see if it's possible to replicate the issue for any potential bugs that may need to be addressed.
|
||||
|
||||
!> We've done our best to stamp out as much flakiness as possible in our end-to-end tests, but it can still occur. If you're unable to reproduce the issue manually, try re-running the test to see if it passes.
|
||||
|
||||
## API Tests
|
||||
|
||||
API Testing is a crucial part of ensuring the smooth functioning of an application. With API testing, we execute a set of operations using the [WooCommerce REST API](https://woocommerce.github.io/woocommerce-rest-api-docs/) and verify that the API responds in an expected, consistent, way. These operations include creating products, customers, and orders through the REST API and then validating the data that we have created.
|
||||
|
||||
The test suite that we use for this process is the [WooCommerce Core API tests](https://github.com/woocommerce/woocommerce/tree/trunk/plugins/woocommerce/tests/api-core-tests). These tests are ran against a store where your extension has been installed and activated. Upon completion of the test, you can check the results either in the Dashboard, or retrieving the test using the QIT CLI, to see if the test has passed or failed. If it fails, you can access an Allure test report that will provide you with a detailed analysis of the reasons for the failure.
|
||||
|
||||
### What to do if it fails
|
||||
|
||||
In the event of a failed API test, you must take the following steps:
|
||||
|
||||
- Review the test report.
|
||||
- Analyze the causes of the failure and attempt to recreate the issue on your local setup.
|
||||
- Resolve the issue and run the test again.
|
||||
- If you believe that the result is incorrect, please do not hesitate to contact us at qit@woocommerce.com.
|
||||
|
||||
## Activation tests
|
||||
|
||||
The Activation test type activates your extension against a freshly created shop and captures any PHP errors or warnings that may occur on activation. This test sets up a WooCommerce store using the provided WordPress and WooCommerce versions. The following statuses will be returned from this test:
|
||||
|
||||
- Success: No PHP notices, warnings, or errors were triggered.
|
||||
- Warning: A non-fatal PHP error was triggered.
|
||||
- Failure: A fatal PHP error was triggered.
|
||||
|
||||
### What to do if it fails
|
||||
|
||||
If your activation test is failing, please take the following steps:
|
||||
|
||||
- Open the test report
|
||||
- Identify the causes of failure. The test will log PHP notices, warnings, and errors that happens when activating your plugin
|
||||
- Try to reproduce it locally and fix the issue
|
||||
- If you think the result is incorrect, please email us at qit@woocommerce.com
|
||||
|
||||
## Security Tests (Beta)
|
||||
|
||||
This test runs an experimental security scanner against a given extension.
|
||||
|
||||
- Success: No security issues errors or warnings.
|
||||
- Warning: Only security issues warnings.
|
||||
- Failure: One or more security issues errors.
|
||||
|
||||
### What Tools Are Used?
|
||||
|
||||
The tools used in the Security Scanner are, currently, [PHPCS](https://github.com/squizlabs/PHP_CodeSniffer) and [SemGrep](https://semgrep.dev/).
|
||||
|
||||
### Can I run it Locally?
|
||||
|
||||
Ideally, you should delegate all the testing execution to QIT. We don't support running the tests outside of QIT, but you can mimick at least the PHPCS rules. The SemGrep rules are not available to be run locally.
|
||||
|
||||
### Which PHPCS Rules Are Enabled?
|
||||
|
||||
Currently, the only rules enabled in the Security Scanner is `WordPress.Security.EscapeOutput`, and `WordPress.Security.ValidatedSanitizedInput`, with some custom sanitizing and escaping functions whitelisted.
|
||||
|
||||
These two rules were chosen as a basic level of security to enforce on all extensions, as they are very accurate and will hardly flag false positives. In the near future, we plan to allow the developer to opt-in to stricter levels of security scans, to show off the good work they do, such as nonce checks, or other security programs, such as generational AI-based scanners that can understand flow of logic, or other third party software other than PHPCS and SemGrep. If you'd like to be a part of the process to define the rules ran against your extensions, feel free to [contact us](https://woocommerce.github.io/qit-documentation/#/contact-us) with your suggestions.
|
||||
|
||||
For a more detailed look on our PHPCS rulesets, please see the sample below.
|
||||
|
||||
```xml
|
||||
<?xml version="1.0"?>
|
||||
<ruleset name="Woo Marketplace PHPCS Security Checks">
|
||||
<description>Woo Marketplace PHPCS Security Checks.</description>
|
||||
|
||||
<arg value="sp"/>
|
||||
<arg name="colors"/>
|
||||
<arg name="extensions" value="php"/>
|
||||
<arg name="parallel" value="8"/>
|
||||
|
||||
<exclude-pattern>tests/*</exclude-pattern>
|
||||
<exclude-pattern>vendor/*</exclude-pattern>
|
||||
<exclude-pattern>vendor-prefixed/*</exclude-pattern>
|
||||
|
||||
|
||||
<rule ref="WordPress.Security.EscapeOutput">
|
||||
<properties>
|
||||
<property name="customEscapingFunctions" type="array" value="WC_Payments_Utils,esc_interpolated_html,wc_help_tip,wc_sanitize_tooltip,wc_selected,wc_kses_notice,wc_esc_json,wc_query_string_form_fields,wc_make_phone_clickable" />
|
||||
</properties>
|
||||
</rule>
|
||||
|
||||
<!-- Lack of nonces will be added at a later point to minimize noise. -->
|
||||
<rule ref="WordPress.Security.NonceVerification">
|
||||
<severity>0</severity>
|
||||
</rule>
|
||||
|
||||
<!-- Do not flag missing "wp_unslash()" calls to globals such as $_POST, and $_GET, etc. -->
|
||||
<rule ref="WordPress.Security.ValidatedSanitizedInput.MissingUnslash">
|
||||
<severity>0</severity>
|
||||
</rule>
|
||||
|
||||
<!-- Do not flag missing checks of "isset" on $_POST, $_GET, etc. -->
|
||||
<rule ref="WordPress.Security.ValidatedSanitizedInput.InputNotValidated">
|
||||
<severity>0</severity>
|
||||
</rule>
|
||||
|
||||
<!-- Do not flag usage of deprecated "// WPCS: XSS ok." -->
|
||||
<rule ref="WordPress.Security.EscapeOutput.DeprecatedWhitelistCommentFound">
|
||||
<severity>0</severity>
|
||||
</rule>
|
||||
|
||||
<!-- Warn about usage of potentially dangerous functions. -->
|
||||
<rule ref="Generic.PHP.ForbiddenFunctions">
|
||||
<properties>
|
||||
<property name="error" value="false" />
|
||||
<property name="forbiddenFunctions" type="array" value="wp_set_auth_cookie=>null,wp_set_current_user=>null"/>
|
||||
</properties>
|
||||
</rule>
|
||||
|
||||
<rule ref="WordPress.Security.PluginMenuSlug"/>
|
||||
|
||||
<!-- When we disable "InputNotValidated", "InputNotSanitized" also gets disabled, and this is not what we want. -->
|
||||
<rule ref="WordPress.Security.ValidatedSanitizedInput.InputNotSanitized">
|
||||
<severity>5</severity>
|
||||
</rule>
|
||||
|
||||
<rule ref="WordPress.Security.ValidatedSanitizedInput">
|
||||
<properties>
|
||||
<property name="customSanitizingFunctions" type="array" value="wc_booking_sanitize_time,wc_clean,wc_sanitize_tooltip,wc_format_decimal,wc_stock_amount,wc_sanitize_permalink,wc_sanitize_textarea" />
|
||||
<property name="customUnslashingSanitizingFunctions" type="array" value="stripslashes"/>
|
||||
</properties>
|
||||
</rule>
|
||||
</ruleset>
|
||||
```
|
||||
|
||||
### What to do When Encountering a Discouraged Function?
|
||||
|
||||
We identify functions that may lead to potential security vulnerabilities and mark them with a Warning using the `Generic.PHP.ForbiddenFunctions.Discouraged` rule.
|
||||
|
||||
While these functions are not inherently unsafe, they frequently contribute to critical vulnerabilities. We flag them to encourage you to review the code for security. If you've confirmed that the code is secure, you can suppress the warning by adding the following comment on the same line as the function: `// phpcs:ignore Generic.PHP.ForbiddenFunctions.Discouraged`
|
||||
|
||||
### What to do if it Fails
|
||||
|
||||
If your security test is failing, please take the following steps:
|
||||
|
||||
- Open the test report
|
||||
- Identify the causes of failure. The test will log any security issues that our scanner identifies
|
||||
- Fix the issue and re-run the test
|
||||
|
||||
### Handling False Positives
|
||||
|
||||
False positives, or alerts for security issues that do not exist in actuality, may occasionally arise during security testing. Though we've chosen PHPCS and SemGrep rules to minimize such occurrences, it's important to address these false positives in a systematic way.
|
||||
|
||||
- **Verification:** The first step is to understand and confirm if it's indeed a false positive. Review the flagged code section and the warning or error raised by the tool. It's always good to revisit the code and the associated rule to understand the potential security implications, if any.
|
||||
- **Report:** If after careful review, the flag still appears to be a false positive, report it to us. Send an email to qit@woocommerce.com explaining the situation, with the specific test result and the corresponding part of your code. Please make sure to include any additional information that can help us understand why you believe it's a false positive.
|
||||
- **Suppression (temporarily):** In the meantime, while we investigate the issue, you might want to suppress the false positive to continue your work without disruption. To do so, add a comment line right before the flagged line in your code.
|
||||
- For PHPCS errors, add `// phpcs:ignore Rule.Name` on the line where the error is reported. Replace `Rule.Name` with the rule that has caused the false positive, eg: `// phpcs:ignore WordPress.Security.ValidatedSanitizedInput`
|
||||
- For SemGrep errors, add `// nosemgrep: rule-id` in a similar manner, replacing rule-id with the SemGrep rule identifier, eg: `// nosemgrep: audit.php.wp.security.xss.query-arg`
|
||||
|
||||
Only suppress the error if you are certain that it is a false positive.
|
||||
|
||||
- **Resolution:** We'll review your report and communicate our findings. If we confirm it's a false positive, we will work towards fine-tuning our rules to prevent such instances in the future. Your cooperation in reporting these instances is invaluable to improve the quality and accuracy of our security testing.
|
||||
|
||||
Please note, suppressing warnings or errors should be done judiciously and is only recommended as a temporary solution. We strongly advise against using it as a permanent solution to avoid security tests. Our goal is to ensure that your extensions are as secure as possible, and ignoring genuine warnings or errors can lead to security vulnerabilities.
|
||||
|
||||
## PHPStan tests
|
||||
|
||||
The PHPStan test type runs level 0 PHPStan checks against your extension. More details on what the rule levels cover can be found in the official PHPStan documentation: [Rule Levels](https://phpstan.org/user-guide/rule-levels).
|
||||
|
||||
### What to do if it fails
|
||||
|
||||
Due to the very nature of WordPress and WooCommerce of not being typed codebases, static code analysis such as PHPStan have a high failure rate when testing WordPress plugins in general.
|
||||
|
||||
This doesn't mean that there are issues with the quality of the code, it's only that the code is not friendly to static code analysis, such as code with extensive usage of getters and setters, which is very common when integrating with WooCommerce Core.
|
||||
|
||||
The PHPStan tests still provide a lot of value for the developers that want to pursue the highest levels of quality. By default, we run PHPStan tests with level 0.
|
||||
|
||||
If your PHPStan test is failing, please take the following steps:
|
||||
|
||||
- You can ignore this test, as we don't use it internally to measure an extensions quality
|
||||
- But if you wish to address the feedback that the static code analsysis provide, open the test report
|
||||
- Identify the causes of failure
|
||||
- Fix the issue and re-run the test
|
||||
- If you think the result is incorrect, please email us at qit@woocommerce.com
|
||||
Loading…
Reference in New Issue