75 lines
3.8 KiB
Markdown
75 lines
3.8 KiB
Markdown
# Monorepo Infrastructure & Tools
|
|
|
|
This document aims to provide an outline for the monorepo's infrastructure as well as the rationale behind the decisions we've made.
|
|
|
|
## Task Orchestration
|
|
|
|
Our repository makes aggressive use of [parallelization using PNPM's `--filter` syntax](https://pnpm.io/filtering). This allows us to minimize the amount of time developers spend waiting for tasks like builds to complete. Each project within the monorepo will contain the following scripts if applicable:
|
|
|
|
```json
|
|
{
|
|
"scripts": {
|
|
"build": "pnpm --if-present --workspace-concurrency=Infinity --stream --filter=\"$npm_package_name...\" '/^build:project:.*$/'",
|
|
"build:project": "pnpm --if-present '/^build:project:.*$/'",
|
|
"lint": "pnpm --if-present '/^lint:lang:.*$/'",
|
|
"lint:fix": "pnpm --if-present '/^lint:fix:lang:.*$/'",
|
|
"watch:build": "pnpm --if-present --workspace-concurrency=Infinity --filter=\"$npm_package_name...\" --parallel '/^watch:build:project:.*$/'",
|
|
"watch:build:project": "pnpm --if-present run '/^watch:build:project:.*$/'"
|
|
}
|
|
}
|
|
```
|
|
|
|
These scripts outline the naming scheme used in order to facilitate [task parallelization using regular expressions](https://pnpm.io/cli/run#running-multiple-scripts). **To ensure consistency across the monorepo, these scripts should not be edited.** New scripts should be added using the naming scheme outlined by the above regular expressions, for example, `build:project:bundle` might be a script to run a tool like `webpack`. We also utilize a number of PNPM options to ensure a positive developer experience:
|
|
|
|
- `--if-present`: Ensures that PNPM will not error if a script is not found.
|
|
- `--workspace-concurrency=Infinity`: Runs as many of the tasks in parallel as possible.
|
|
- `--stream`: Makes the script output legible by putting all of their output into a single stream.
|
|
- `--filter="$npm_package_name..."`: This filter tells PNPM that we want to run the script against the current project _and_ all of its dependencies down the graph (dependencies first).
|
|
|
|
To further improve the build times, we used two additional techniques:
|
|
|
|
- In the case of the `build` script, we offer both building packages with (`build`) and without (`build:project`) its dependencies.
|
|
- Using `wireit`-based task output caching (details are below).
|
|
|
|
## Task Output Caching
|
|
|
|
Our repository uses [`wireit`](https://github.com/google/wireit) to provide task output caching. In particular, this allows us to cache the output of `build` scripts so that they don't run unnecessarily.
|
|
The goal is to minimize the amount of time that developers spend waiting for projects to build.
|
|
|
|
```json
|
|
{
|
|
"scripts": {
|
|
"build": "pnpm --if-present --workspace-concurrency=Infinity --stream --filter=\"$npm_package_name...\" '/^build:project:.*$/'",
|
|
"build:project": "pnpm --if-present '/^build:project:.*$/'",
|
|
"build:project:bundle": "wireit"
|
|
},
|
|
"wireit": {
|
|
"build:project:bundle": {
|
|
"command": "webpack",
|
|
"files": [
|
|
"... - package resources as input"
|
|
],
|
|
"output": [
|
|
"... - package resources as output"
|
|
],
|
|
"dependencies": [
|
|
"dependencyOutputs"
|
|
]
|
|
},
|
|
"dependencyOutputs": {
|
|
"files": [
|
|
"... - dependencies resources as input",
|
|
"... - updated automatically by monorepo tooling which hooks up into `pnpm install`",
|
|
"... - see `.pnpmfile.cjs` file and https://pnpm.io/pnpmfile for details"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
In the example above, `build:project:bundle` invokes `wireit`, which conditionally executes `webpack` (based on the state of resources).
|
|
A simplified take on `wireit` is the following:
|
|
|
|
- if input sources are changed or output sources are not generated yet, `wireit` will execute `webpack` command and cache output sources
|
|
- if input sources are unchanged, `wireit` will create output sources from their cached version
|