docs(nx-cloud): improve nx agents docs with more details (#20531)
This commit is contained in:
Juri Strumpflohner
GitHub
parent
9e322e8620
commit
4c17667c84
@ -1,53 +1,141 @@
|
|||||||
# Nx Agents: The Next Leap in Distributed Task Execution
|
# Nx Agents: The Next Leap in Distributed Task Execution
|
||||||
|
|
||||||
Nx Agents represent the next evolution of [Nx Cloud's Distributed Task Execution (DTE)](/ci/features/distribute-task-execution), bringing a new level of efficiency and simplicity to your CI/CD pipelines. Features include dynamic scaling, flaky task re-running, and intelligent task splitting and distribution. Keep reading to learn more.
|
{% callout type="note" title="Early Preview Doc - Subject to Change" %}
|
||||||
|
**Early Preview of Nx Agents:** This is a work-in-progress feature, with a public launch anticipated in Feb 2024. Keep an eye on this document for continuous updates. Interested in early access? [Sign up here](https://go.nx.dev/nx-agents-ea).
|
||||||
|
{% /callout %}
|
||||||
|
|
||||||
{% youtube
|
{% youtube
|
||||||
src="https://youtu.be/KPCMg_Dn0Eo?si=TK-Ky8270ZGNmMUc"
|
src="https://youtu.be/KPCMg_Dn0Eo?si=TK-Ky8270ZGNmMUc"
|
||||||
title="Nx Agents in action splitting e2e tests at a file level"
|
title="Nx Agents in action splitting e2e tests at a file level"
|
||||||
/%}
|
/%}
|
||||||
|
|
||||||
Currently in private beta, Nx Agents are slated for public release in Q1 2024. Don't miss the opportunity to be among the first to experience this groundbreaking tool. Sign up now for early access.
|
Nx Agents represent the next evolution of [Nx Cloud's Distributed Task Execution (DTE)](/ci/features/distribute-task-execution), bringing a new level of efficiency and simplicity to your CI/CD pipelines. It takes away the complexity of configuring agents and comes with features such as scaling of agents based on the PR, flaky task re-running, and intelligent task splitting and distribution. Keep reading to learn more.
|
||||||
|
|
||||||
{% call-to-action title="Sign Up for Early Access" icon="nxcloud" description="Experience Nx Agents for yourself" url="https://cloud.nx.app/workflows-early-access" /%}
|
Currently in private beta, Nx Agents are slated for public release in February 2024. Don't miss the opportunity to be among the first to experience this groundbreaking tool. Sign up now for early access.
|
||||||
|
|
||||||
## Seamless Integration
|
{% call-to-action title="Sign Up for Early Access" icon="nxcloud" description="Experience Nx Agents for yourself" url="https://go.nx.dev/nx-agents-ea" /%}
|
||||||
|
|
||||||
Enabling task distribution with Nx Agents can be done in a single line. Simply add the following to your CI pipeline configuration:
|
## What's the Difference to DTE?
|
||||||
|
|
||||||
|
Nx Cloud's [Distributed Task Execution (DTE)](/ci/features/distribute-task-execution) introduced an easy way to intelligently distribute tasks across machines, allowing for a more fine-grained distribution taking historical data as well as the task dependencies into account.
|
||||||
|
|
||||||
|
Using DTE you have to configure and instantiate your agents, which might be more or less complex depending on your CI provider. We have some guides on how to do that [here](/ci/recipes/set-up).
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Nx Agents take away that complexity by delegating the agent management to Nx Cloud. You can think of them as a managed version of DTE.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Keep reading to learn what the configuration and setup looks like.
|
||||||
|
|
||||||
|
## Managed Agents, Seamless Configuration
|
||||||
|
|
||||||
|
Enabling task distribution with Nx Agents can be done in a single line. Simply add the following to your CI pipeline configuration before your other steps:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
- name: Start CI run
|
- name: Start CI run
|
||||||
run: 'npx nx-cloud start-ci-run --distributes-on="8 linux-medium"'
|
run: 'npx nx-cloud start-ci-run --distributes-on="8 linux-medium-js"'
|
||||||
...
|
...
|
||||||
```
|
```
|
||||||
|
|
||||||
Continue with your standard CI steps thereafter.
|
This instructs Nx Cloud to distribute tasks across 8 agents of type `linux-medium-js`. `linux-medium-js` is the name of the launch template that will be used to provision the agent. The default launch templates [can be found here](https://github.com/nrwl/nx-cloud-workflows/blob/main/launch-templates/linux.yaml) (there will be more once Nx Agents is publicly available).
|
||||||
|
|
||||||
|
You can also define your own "launch templates" (here's an [example from the Nx repo](https://github.com/nrwl/nx/blob/master/.nx/workflows/agents.yaml)):
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
...
|
# .nx/workflows/agents.yaml
|
||||||
- run: yarn install --frozen-lockfile
|
launch-templates:
|
||||||
- name: Run verification
|
linux-medium:
|
||||||
run: 'npx nx affected -t build test lint && npx nx affected -t e2e-ci --parallel 1'
|
resource-class: 'docker_linux_amd64/medium+'
|
||||||
```
|
env:
|
||||||
|
CI: 'true'
|
||||||
|
GIT_AUTHOR_EMAIL: test@test.com
|
||||||
|
...
|
||||||
|
NX_CLOUD_ACCESS_TOKEN: '{{secrets.NX_CLOUD_ACCESS_TOKEN}}'
|
||||||
|
init-steps:
|
||||||
|
...
|
||||||
|
- name: Install Pnpm
|
||||||
|
script: |
|
||||||
|
npm install -g @pnpm/exe@8.7.4
|
||||||
|
|
||||||
`linux-medium` is the name of the launch template that will be used to provision the agent. The default launch templates [can be found here](https://github.com/nrwl/nx-cloud-workflows/blob/main/launch-templates/linux.yaml). You can can also define your own template as can be [seen in the Nx codebase](https://github.com/nrwl/nx/blob/master/.nx/workflows/agents.yaml).
|
- name: Pnpm Install
|
||||||
|
script: |
|
||||||
|
pnpm install --frozen-lockfile
|
||||||
|
|
||||||
|
- name: Install Cypress
|
||||||
|
script: pnpm exec cypress install
|
||||||
|
|
||||||
|
- name: Install Rust
|
||||||
|
- ...
|
||||||
|
```
|
||||||
|
|
||||||
## Intelligent Dynamic Scaling
|
## Intelligent Dynamic Scaling
|
||||||
|
|
||||||
While [Nx Cloud's DTE](/ci/features/distribute-task-execution) requires static configuration of agents, often necessitating adjustments as your workspace evolves, Nx Agents introduce a dynamic scaling feature. This innovative approach automatically scales your CI resources, spinning up or tearing down agents in response to the specific demands of your pull requests (PRs).
|
Instead of defining
|
||||||
|
|
||||||
## Revolutionizing Task Splitting
|
```
|
||||||
|
--distributes-on="8 linux-medium-js"
|
||||||
|
```
|
||||||
|
|
||||||
Consider an e2e project utilizing Cypress or Playwright. To leverage [affected](/ci/features/affected), [caching](/ci/features/remote-cache) and [distribution](/ci/features/distribute-task-execution), traditionally, you would need to break down your project into smaller, more manageable units. However, this can lead to a tedious and less efficient developer experience.
|
...which always runs tasks on the same amount of machines, you can also have Nx Cloud scale the number of agents based on the size of your PR.
|
||||||
|
|
||||||
Nx Agents, in conjunction with a forthcoming Nx feature, solve this challenge by being able to dividing tasks (like e2e) into smaller, more refined units (e.g., at the file/test level). This enhances distribution across dynamically provisioned Nx Agents and can drastically reduce CI time.
|
```yaml {% fileName=".nx/workflows/dynamic-changesets.yaml" %}
|
||||||
|
distributes-on:
|
||||||
|
small-changeset: 1 linux-medium
|
||||||
|
medium-changeset: 6 linux-medium
|
||||||
|
large-changeset: 10 linux-medium
|
||||||
|
```
|
||||||
|
|
||||||
|
{% callout type="note" title="How is the size of the PR determined?" %}
|
||||||
|
To determine the size of the PR, Nx Cloud calculates the relationship between the number of [affected projects](/ci/features/affected) and the total number of projects in the workspace. It then assigns it to one of the three categories: small, medium, or large. This calculation is static right now but might be configurable once Nx Agents is publicly available.
|
||||||
|
{% /callout %}
|
||||||
|
|
||||||
|
You can then reference it in your CI pipeline configuration:
|
||||||
|
|
||||||
|
```yaml {% fileName=".github/workflows/main.yaml" %}
|
||||||
|
...
|
||||||
|
jobs:
|
||||||
|
- job: main
|
||||||
|
displayName: Main Job
|
||||||
|
...
|
||||||
|
steps:
|
||||||
|
- checkout: self
|
||||||
|
persistCredentials: true
|
||||||
|
- script: |
|
||||||
|
npx nx-cloud start-ci-run --distributes-on=".nx/workflows/dynamic-changesets.yaml" --stop-agents-after="e2e-wrapper"
|
||||||
|
displayName: Start CI run
|
||||||
|
- ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Automatic Task Splitting
|
||||||
|
|
||||||
|
Imagine you're working on an end-to-end (e2e) project using tools like Cypress or Playwright. Traditionally, to make the most of features like [affected](/ci/features/affected), [caching](/ci/features/remote-cache), and [distribution](/ci/features/distribute-task-execution), you'd need to divide your project into smaller parts. But this approach can often be cumbersome and less efficient for developers.
|
||||||
|
|
||||||
|
Nx is on the verge of introducing a game-changing feature that enables dynamic target definitions for projects (more details to come). Paired with Nx Agents, this innovation allows you to distribute e2e tests at the file level across various agents.
|
||||||
|
|
||||||
|
This significantly cuts down the time required to run e2e tests. For instance, in the video shown at the beginning of the page, e2e test durations plummeted from 90 minutes to just 10 minutes.
|
||||||
|
|
||||||
## Flaky Task Re-Running: Enhancing Reliability
|
## Flaky Task Re-Running: Enhancing Reliability
|
||||||
|
|
||||||
Flaky tasks can be a persistent challenge in software development. With Nx Agents, we’ve developed a reliable method to detect and automatically re-run such tasks. Utilizing cache hash keys to identify tasks and leveraging data from previous runs on Nx Cloud, Nx Agents can re-run flaky tasks on different agents, up to a predetermined maximum number of retries. This process is entirely transparent to the user, ensuring a smooth and reliable CI/CD experience.
|
Flaky tasks are a common headache, particularly with tests and end-to-end (e2e) tests. Nx Agents offer a solid solution to detect and automatically retry such unreliable tasks.
|
||||||
|
|
||||||
|
Nx Cloud keeps track of the targets being executed. A task, like `myapp:e2e`, is labeled as flaky **if it shows different outcomes (status codes) for the same cache hash key**. This key is an ideal task identifier, encompassing the command, environment variables, source files, and more.
|
||||||
|
|
||||||
|
Consider this scenario:
|
||||||
|
|
||||||
|
- An Nx agent runs `myapp:e2e`.
|
||||||
|
- Nx calculates the cache hash key for this task.
|
||||||
|
- **`myapp:e2e` fails**; Nx Cloud notes this failure along with the cache key.
|
||||||
|
- In a subsequent run, `myapp:e2e` is executed again.
|
||||||
|
- Nx recalculates the cache hash key, which matches the previous run's key (no existing cache since the initial task failed).
|
||||||
|
- This time, `myapp:e2e` **succeeds**.
|
||||||
|
- Nx Cloud identifies the task as flaky and stores this data temporarily.
|
||||||
|
|
||||||
|
As a result, if Nx Cloud has marked a task as flaky, it will be automatically retried, ideally on a different Nx Agent to prevent issues from any residues of earlier runs.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
Sign up now for early access and be one of the first to try Nx Agents.
|
Sign up now for early access and be one of the first to try Nx Agents.
|
||||||
|
|
||||||
{% call-to-action title="Sign Up for Early Access" icon="nxcloud" description="Experience Nx Cloud Agents for yourself" url="https://cloud.nx.app/workflows-early-access" /%}
|
{% call-to-action title="Sign Up for Early Access" icon="nxcloud" description="Experience Nx Cloud Agents for yourself" url="https://go.nx.dev/nx-agents-ea" /%}
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user