Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(core): add azure and gcs powerpack plugins #28668

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions docs/external-generated/packages-metadata.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,27 @@
[
{
"description": "A Nx Powerpack plugin which provides a Nx cache which can be self hosted on Azure Blob Storage.",
"documents": [
{
"id": "overview",
"name": "Overview",
"description": "A Nx Powerpack plugin which provides a Nx cache which can be self hosted on Azure Blob Storage.",
"file": "external-generated/packages/powerpack-azure-cache/documents/overview",
"itemList": [],
"isExternal": false,
"path": "powerpack-azure-cache/documents/overview",
"tags": [],
"originalFilePath": "shared/packages/powerpack-azure-cache/powerpack-azure-cache-plugin"
}
],
"executors": [],
"generators": [],
"githubRoot": "https://github.com/nrwl/nx/blob/master",
"name": "powerpack-azure-cache",
"packageName": "@nx/powerpack-azure-cache",
"root": "/libs/nx-packages/powerpack-azure-cache",
"source": "/libs/nx-packages/powerpack-azure-cache/src"
},
{
"description": "A Nx Powerpack plugin which allows users to write and apply rules for your entire workspace that help with consistency, maintainability, reliability and security.",
"documents": [
Expand Down Expand Up @@ -32,6 +55,50 @@
"root": "/libs/nx-packages/powerpack-conformance",
"source": "/libs/nx-packages/powerpack-conformance/src"
},
{
"description": "A Nx Powerpack plugin which is specific to Nx Enterprise Cloud workspaces.",
"documents": [],
"executors": [],
"generators": [
{
"description": "Initialize Nx Powerpack Enterprise Cloud config",
"file": "external-generated/packages/powerpack-enterprise-cloud/generators/init.json",
"hidden": false,
"name": "init",
"originalFilePath": "/libs/nx-packages/powerpack-enterprise-cloud/src/generators/init/schema.json",
"path": "powerpack-enterprise-cloud/generators/init",
"type": "generator"
}
],
"githubRoot": "https://github.com/nrwl/nx/blob/master",
"name": "powerpack-enterprise-cloud",
"packageName": "@nx/powerpack-enterprise-cloud",
"root": "/libs/nx-packages/powerpack-enterprise-cloud",
"source": "/libs/nx-packages/powerpack-enterprise-cloud/src"
},
{
"description": "A Nx Powerpack plugin which provides a Nx cache which can be self hosted on Google Cloud Storage.",
"documents": [
{
"id": "overview",
"name": "Overview",
"description": "A Nx Powerpack plugin which provides a Nx cache which can be self hosted on Google Cloud Storage.",
"file": "external-generated/packages/powerpack-gcs-cache/documents/overview",
"itemList": [],
"isExternal": false,
"path": "powerpack-gcs-cache/documents/overview",
"tags": [],
"originalFilePath": "shared/packages/powerpack-gcs-cache/powerpack-gcs-cache-plugin"
}
],
"executors": [],
"generators": [],
"githubRoot": "https://github.com/nrwl/nx/blob/master",
"name": "powerpack-gcs-cache",
"packageName": "@nx/powerpack-gcs-cache",
"root": "/libs/nx-packages/powerpack-gcs-cache",
"source": "/libs/nx-packages/powerpack-gcs-cache/src"
},
{
"description": "Package to provide the ability to activate and read licenses for Nx Powerpack.",
"documents": [],
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
title: Overview of the Nx powerpack-azure-cache Plugin
description: The powerpack-azure-cache Nx plugin enables you to use an Amazon S3 bucket to host your remote cache instead of Nx Cloud
---

The `@nx/powerpack-azure-cache` plugin enables you to use an [Amazon S3](https://aws.amazon.com/azure) bucket instead of Nx Cloud to host your remote cache.

This plugin will enable the remote cache for your Nx workspace, but does not provide any of the other features of Nx Cloud. If you want to leverage [distributed task execution](/ci/features/distribute-task-execution), [re-running flaky tasks](/ci/features/flaky-tasks) or [automatically splitting tasks](/ci/features/split-e2e-tasks), you'll need to [connect to Nx Cloud](/ci/intro/connect-to-nx-cloud) and use [Nx Replay](/ci/features/remote-cache) instead.

{% callout type="warning" title="Potential Cache Poisoning" %}
Using your own Amazon S3 bucket to host the remote cache opens you up to the possibility of [cache poisoning](/troubleshooting/unknown-local-cache). To avoid this, use [Nx Replay](/ci/features/remote-cache).
{% /callout %}

{% callout title="This plugin requires an active Nx Powerpack license" %}
In order to use `@nx/powerpack-azure-cache`, you need to have an active Powerpack license. If you don't have a license or it has expired, your cache will no longer be shared and each machine will use its local cache.
{% /callout %}

## Set Up @nx/powerpack-azure-cache

### 1. Install the Package

1. [Activate Powerpack](/nx-enterprise/activate-powerpack) if you haven't already
2. Install the package

```shell
nx add @nx/powerpack-azure-cache
```

### 2. Authenticate with Google Cloud

There are four different ways to authenticate with Google Cloud. They will be attempted in this order:

1. Environment variables
2. INI config files
3. Single sign-on
4. `nx.json` settings

#### Environment Variables

[Google Cloud provides environment variables](https://docs.aws.amazon.com/sdkref/latest/guide/environment-variables.html) that can be used to authenticate:

| **Environment Variable** | **Description** |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AWS_ACCESS_KEY_ID` | The access key for your AWS account. |
| `AWS_SECRET_ACCESS_KEY` | The secret key for your AWS account. |
| `AWS_SESSION_TOKEN` | The session key for your AWS account. This is only needed when you are using temporary credentials. |
| `AWS_CREDENTIAL_EXPIRATION` | The expiration time of the credentials contained in the environment variables described above. This value must be in a format compatible with the [ISO-8601 standard](https://en.wikipedia.org/wiki/ISO_8601) and is only needed when you are using temporary credentials. |

Both the `AWS_ACCESS_KEY_ID` and the `AWS_SECRET_ACCESS_KEY` environment variables are required to use the environment variable authentication method.

Here's an example of using OICD in GitHub Actions to set the environment variables in CI:

```yaml {% fileName=".github/workflows/ci.yml" %}
name: CI
...
permissions:
id-token: write
...

env:
NX_DB_CACHE: true

jobs:
main:
runs-on: ubuntu-latest
steps:
...

- name: 'Configure AWS Credentials'
uses: aws-actions/[email protected]
with:
role-to-assume: arn:aws:iam::123456789123:role/GhAIBucketUserRole
aws-region: us-east-1

...

- run: pnpm exec nx affected -t lint test build
```

#### INI Config Files

AWS can read your authentication credentials from [shared INI config files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html). The files are located at `~/.aws/credentials` and `~/.aws/config`. Both files are expected to be INI formatted with section names corresponding to profiles. Sections in the credentials file are treated as profile names, whereas profile sections in the config file must have the format of `[profile profile-name]`, except for the default profile. Profiles that appear in both files will not be merged, and the version that appears in the credentials file will be given precedence over the profile found in the config file.

#### Single Sign-On

Nx can read the active access token [created after running `aws sso login`](https://docs.aws.amazon.com/sdkref/latest/guide/understanding-sso.html) then request temporary AWS credentials. You can create the `AwsCredentialIdentityProvider` functions using the inline SSO parameters (`ssoStartUrl`, `ssoAccountId`, `ssoRegion`, `ssoRoleName`) or load them from [AWS SDKs and Tools shared configuration and credentials files](https://docs.aws.amazon.com/credref/latest/refdocs/creds-config-files.html). Profiles in the `credentials` file are given precedence over profiles in the `config` file.

#### Credentials in `nx.json` File

Storing your credentials in the `nx.json` file is the least secure of the 4 authentication options, since anyone with read access to your code base will have access to your AWS credentials.

```jsonc {% fileName="nx.json" %}
{
"azure": {
"ssoProfile": "default",
"accessKeyId": "MYACCESSKEYID",
"secretAccessKey": "MYSECRETACCESSKEY"
}
}
```

| **Property** | **Description** |
| ------------------- | ----------------------------------------------------------------------------- |
| **ssoProfile** | The name of the profile to use from your AWS CLI SSO Configuration (optional) |
| **endpoint** | The AWS endpoint URL (optional) |
| **accessKeyId** | AWS Access Key ID (optional) |
| **secretAccessKey** | AWS secret access key (optional) |

### 3. Configure S3 Cache

Regardless of how you manage your AWS authentication, you need to configure your Nx cache in the `nx.json` file. The `bucket` that you specify needs to already exist - Nx doesn't create it for you.

```jsonc {% fileName="nx.json" %}
{
"azure": {
"region": "us-east-1",
"bucket": "my-bucket",
"encryptionKey": "create-your-own-key"
}
}
```

| **Property** | **Description** |
| ----------------- | --------------------------------------------------------------------------------- |
| **region** | The id of the AWS region to use |
| **bucket** | The name of the AWS bucket to use |
| **encryptionKey** | Nx encryption key used to encrypt and decrypt artifacts from the cache (optional) |
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "init",
"factory": "./src/generators/init/init",
"schema": {
"$schema": "http://json-schema.org/schema",
"id": "NxPowerpackEnterpriseCloudInit",
"title": "Add Powerpack Enterprise Cloud plugin to the workspace",
"type": "object",
"cli": "nx",
"properties": {},
"additionalProperties": false,
"required": [],
"presets": []
},
"description": "Initialize Nx Powerpack Enterprise Cloud config",
"implementation": "/libs/nx-packages/powerpack-enterprise-cloud/src/generators/init/init.ts",
"aliases": [],
"hidden": false,
"path": "/libs/nx-packages/powerpack-enterprise-cloud/src/generators/init/schema.json",
"type": "generator"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
title: Overview of the Nx powerpack-gcs-cache Plugin
description: The powerpack-gcs-cache Nx plugin enables you to use an Amazon S3 bucket to host your remote cache instead of Nx Cloud
---

The `@nx/powerpack-gcs-cache` plugin enables you to use an [Amazon S3](https://aws.amazon.com/gcs) bucket instead of Nx Cloud to host your remote cache.

This plugin will enable the remote cache for your Nx workspace, but does not provide any of the other features of Nx Cloud. If you want to leverage [distributed task execution](/ci/features/distribute-task-execution), [re-running flaky tasks](/ci/features/flaky-tasks) or [automatically splitting tasks](/ci/features/split-e2e-tasks), you'll need to [connect to Nx Cloud](/ci/intro/connect-to-nx-cloud) and use [Nx Replay](/ci/features/remote-cache) instead.

{% callout type="warning" title="Potential Cache Poisoning" %}
Using your own Amazon S3 bucket to host the remote cache opens you up to the possibility of [cache poisoning](/troubleshooting/unknown-local-cache). To avoid this, use [Nx Replay](/ci/features/remote-cache).
{% /callout %}

{% callout title="This plugin requires an active Nx Powerpack license" %}
In order to use `@nx/powerpack-gcs-cache`, you need to have an active Powerpack license. If you don't have a license or it has expired, your cache will no longer be shared and each machine will use its local cache.
{% /callout %}

## Set Up @nx/powerpack-gcs-cache

### 1. Install the Package

1. [Activate Powerpack](/nx-enterprise/activate-powerpack) if you haven't already
2. Install the package

```shell
nx add @nx/powerpack-gcs-cache
```

### 2. Authenticate with Google Cloud

There are four different ways to authenticate with Google Cloud. They will be attempted in this order:

1. Environment variables
2. INI config files
3. Single sign-on
4. `nx.json` settings

#### Environment Variables

[Google Cloud provides environment variables](https://docs.aws.amazon.com/sdkref/latest/guide/environment-variables.html) that can be used to authenticate:

| **Environment Variable** | **Description** |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AWS_ACCESS_KEY_ID` | The access key for your AWS account. |
| `AWS_SECRET_ACCESS_KEY` | The secret key for your AWS account. |
| `AWS_SESSION_TOKEN` | The session key for your AWS account. This is only needed when you are using temporary credentials. |
| `AWS_CREDENTIAL_EXPIRATION` | The expiration time of the credentials contained in the environment variables described above. This value must be in a format compatible with the [ISO-8601 standard](https://en.wikipedia.org/wiki/ISO_8601) and is only needed when you are using temporary credentials. |

Both the `AWS_ACCESS_KEY_ID` and the `AWS_SECRET_ACCESS_KEY` environment variables are required to use the environment variable authentication method.

Here's an example of using OICD in GitHub Actions to set the environment variables in CI:

```yaml {% fileName=".github/workflows/ci.yml" %}
name: CI
...
permissions:
id-token: write
...

env:
NX_DB_CACHE: true

jobs:
main:
runs-on: ubuntu-latest
steps:
...

- name: 'Configure AWS Credentials'
uses: aws-actions/[email protected]
with:
role-to-assume: arn:aws:iam::123456789123:role/GhAIBucketUserRole
aws-region: us-east-1

...

- run: pnpm exec nx affected -t lint test build
```

#### INI Config Files

AWS can read your authentication credentials from [shared INI config files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html). The files are located at `~/.aws/credentials` and `~/.aws/config`. Both files are expected to be INI formatted with section names corresponding to profiles. Sections in the credentials file are treated as profile names, whereas profile sections in the config file must have the format of `[profile profile-name]`, except for the default profile. Profiles that appear in both files will not be merged, and the version that appears in the credentials file will be given precedence over the profile found in the config file.

#### Single Sign-On

Nx can read the active access token [created after running `aws sso login`](https://docs.aws.amazon.com/sdkref/latest/guide/understanding-sso.html) then request temporary AWS credentials. You can create the `AwsCredentialIdentityProvider` functions using the inline SSO parameters (`ssoStartUrl`, `ssoAccountId`, `ssoRegion`, `ssoRoleName`) or load them from [AWS SDKs and Tools shared configuration and credentials files](https://docs.aws.amazon.com/credref/latest/refdocs/creds-config-files.html). Profiles in the `credentials` file are given precedence over profiles in the `config` file.

#### Credentials in `nx.json` File

Storing your credentials in the `nx.json` file is the least secure of the 4 authentication options, since anyone with read access to your code base will have access to your AWS credentials.

```jsonc {% fileName="nx.json" %}
{
"gcs": {
"ssoProfile": "default",
"accessKeyId": "MYACCESSKEYID",
"secretAccessKey": "MYSECRETACCESSKEY"
}
}
```

| **Property** | **Description** |
| ------------------- | ----------------------------------------------------------------------------- |
| **ssoProfile** | The name of the profile to use from your AWS CLI SSO Configuration (optional) |
| **endpoint** | The AWS endpoint URL (optional) |
| **accessKeyId** | AWS Access Key ID (optional) |
| **secretAccessKey** | AWS secret access key (optional) |

### 3. Configure S3 Cache

Regardless of how you manage your AWS authentication, you need to configure your Nx cache in the `nx.json` file. The `bucket` that you specify needs to already exist - Nx doesn't create it for you.

```jsonc {% fileName="nx.json" %}
{
"gcs": {
"region": "us-east-1",
"bucket": "my-bucket",
"encryptionKey": "create-your-own-key"
}
}
```

| **Property** | **Description** |
| ----------------- | --------------------------------------------------------------------------------- |
| **region** | The id of the AWS region to use |
| **bucket** | The name of the AWS bucket to use |
| **encryptionKey** | Nx encryption key used to encrypt and decrypt artifacts from the cache (optional) |
Loading