diff --git a/.github/workflows/initialize.yml b/.github/workflows/initialize.yml deleted file mode 100644 index 38b6695..0000000 --- a/.github/workflows/initialize.yml +++ /dev/null @@ -1,56 +0,0 @@ -name: Template Repository Initialization - -on: - # Triggers the workflow on creation of repository - create: - workflow_dispatch: - -env: - # Use docker.io for Docker Hub if empty - REGISTRY: ghcr.io - # github.repository as / - IMAGE_NAME: ${{ github.repository }} - -permissions: - contents: write - -jobs: - initialize_repository: - if: github.repository != 'FAIRmat-NFDI/nomad-distribution-template' - runs-on: ubuntu-latest - - steps: - - uses: actions/checkout@v3 - - # Replaces the template repository name in the README with the new one - - name: Update README - run: | - sed -i "s|GITHUB_REPOSITORY_OWNER|${{ github.repository_owner }}|g" template_README.md - sed -i "s|GITHUB_REPOSITORY_NAME|${{ github.event.repository.name }}|g" template_README.md - sed -i "s|GITHUB_REPOSITORY|${{ github.repository }}|g" template_README.md - sed -i "s|GITHUB_REPOSITORY_NAME|${{ github.event.repository.name }}|g" template_README.md - mv template_README.md README.md - - # Replaces the template repository name in the docker config file with the new one - - name: Update docker-compose.yaml - run: | - export image_name="${{ github.repository }}" - sed -i "s|fairmat-nfdi/nomad-distribution-template|${image_name,,}|g" docker-compose.yaml - - # Replaces the template repository name in the nomad config with the new one - - name: Update nomad.yaml - run: | - export image_name="${{ github.repository }}" - sed -i "s|fairmat-nfdi/nomad-distribution-template|${image_name,,}|g" configs/nomad.yaml - - # Deletes this workflow file to prevent it from running on branch creation - - name: Delete initialization workflow - run: rm .github/workflows/initialize.yml - - # Commits all changes - - name: Commit repository initialization - run: | - git config --global user.name github-actions - git config --global user.email github-actions@github.com - git commit -am "Repository initialization" - git push diff --git a/README.md b/README.md index 8195619..9070ee4 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,164 @@ -![docker image](https://github.com/FAIRmat-NFDI/nomad-distribution-template/actions/workflows/docker-publish.yml/badge.svg) - -> [!IMPORTANT] -> The templated repository will run a GitHub action on creation which might take a few minutes. -> After the workflow finishes you should refresh the page and this message should disappear. -> If this message persists you might need to trigger the workflow manually by navigating to the -> "Actions" tab at the top, clicking "Template Repository Initialization" on the left side, -> and triggering it by clicking "Run workflow" under the "Run workflow" button on the right. - -# NOMAD Oasis Distribution *Template* -This repository is a template for creating your own custom NOMAD Oasis distribution image. -Click [here](https://github.com/new?template_name=nomad-distribution-template&template_owner=FAIRmat-NFDI) -to use this template, or click the `Use this template` button in the upper right corner of -the main GitHub page for this template. +![docker image](https://github.com/blueraft/nomad-distribution-template/actions/workflows/docker-publish.yml/badge.svg) + +# blueraft's NOMAD Oasis Distribution + +This is the NOMAD Oasis distribution of blueraft. +Below are instructions for how to [deploy this distribution](#deploying-the-distribution) +and how to customize it through [adding plugins](#adding-a-plugin). + +> [!IMPORTANT] +> Depending on the settings of the owner of this repository, the distributed image might +> be private and require authentication to pull. +> If you want to keep the image private you need to configure and use a personal access +> token (PAT) according to the instructions in the GitHub docs [here](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic). +> If you want to make the image public (recommended), you should make sure that your +> organization settings allow public packages and make this package public after building it. +> You can read more about this in the GitHub docs [here](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility). + +> [!TIP] +> In order for others to find and learn from your distribution we in FAIRmat would +> greatly appreciate it if you would add the topic `nomad-distribution` by clicking the +> ⚙️ next to "About" on the main GitHub page for this repository. + +## Deploying the distribution + +Below are instructions for how to deploy this NOMAD Oasis distribution +[for a new Oasis](#for-a-new-oasis) and [for an existing Oasis](#for-an-existing-oasis) + +### For a new Oasis + +1. Make sure you have [docker](https://docs.docker.com/engine/install/) installed. + Docker nowadays comes with `docker compose` built in. Prior, you needed to + install the stand-alone [docker-compose](https://docs.docker.com/compose/install/). + +2. Clone the repository or download the repository as a zip file. + +```sh +git clone https://github.com/blueraft/nomad-distribution-template.git +cd nomad-distribution-template +``` + +or + +```sh +curl-L -o nomad-distribution-template.zip "https://github.com/blueraft/nomad-distribution-template/archive/main.zip" +unzip nomad-distribution-template.zip +cd nomad-distribution-template +``` + +3. _On Linux only,_ recursively change the owner of the `.volumes` directory to the nomad user (1000) + +```sh +sudo chown -R 1000 .volumes +``` + +4. Pull the images specified in the `docker-compose.yaml` + +Note that the image needs to be public or you need to provide a PAT (see "Important" note above). + +```sh +docker compose pull +``` + +5. And run it with docker compose in detached (--detach or -d) mode + +```sh +docker compose up -d +``` + +6. Optionally you can now test that NOMAD is running with + +``` +curl localhost/nomad-oasis/alive +``` + +7. Finally, open [http://localhost/nomad-oasis](http://localhost/nomad-oasis) in your browser to start using your new NOMAD Oasis. + +Whenever you update your image you need to shut down NOMAD using + +```sh +docker compose down +``` + +and then repeat steps 4. and 5. above. + +#### NOMAD Remote Tools Hub (NORTH) + +To run NORTH (the NOMAD Remote Tools Hub), the `hub` container needs to run docker and +the container has to be run under the docker group. You need to replace the default group +id `991` in the `docker-compose.yaml`'s `hub` section with your systems docker group id. +Run `id` if you are a docker user, or `getent group | grep docker` to find your +systems docker gid. The user id 1000 is used as the nomad user inside all containers. + +You can find more details on setting up and maintaining an Oasis in the NOMAD docs here: +[nomad-lab.eu/prod/v1/docs/oasis/install.html](https://nomad-lab.eu/prod/v1/docs/oasis/install.html) + +### For an existing Oasis + +If you already have an Oasis running you only need to change the image being pulled in +your `docker-compose.yaml` with `ghcr.io/blueraft/nomad-distribution-template:main` for the services +`worker`, `app`, `north`, and `logtransfer`. + +If you want to use the `nomad.yaml` from this repository you also need to comment out +the inclusion of the `nomad.yaml` under the volumes key of those services in the +`docker-compose.yaml`. + +```yaml +volumes: + # - ./configs/nomad.yaml:/app/nomad.yaml +``` + +To run the new image you can follow steps 5. and 6. [above](#for-a-new-oasis). + +## Adding a plugin + +To add a new plugin to the docker image you should add it to the plugins table in the `pyproject.toml` file. + +Here you can put either plugins distributed to PyPI, e.g. + +``` +nomad-material-processing +``` + +or plugins in a git repository with either the commit hash + +``` +"nomad-measurements @ git+https://github.com/FAIRmat-NFDI/nomad-measurements.git@71b7e8c9bb376ce9e8610aba9a20be0b5bce6775" +``` + +or with a tag + +``` +"nomad-measurements @ git+https://github.com/FAIRmat-NFDI/nomad-measurements.git@v0.0.4" +``` + +To add a plugin in a subdirectory of a git repository you can use the `subdirectory` option, e.g. + +``` +"ikz_pld_plugin @ git+https://github.com/FAIRmat-NFDI/AreaA-data_modeling_and_schemas.git@30fc90843428d1b36a1d222874803abae8b1cb42#subdirectory=PVD/PLD/jeremy_ikz/ikz_pld_plugin" +``` + +If the plugin is new, you also need to add it under `plugins` in the [nomad.yaml](nomad.yaml) +config file that will be included in the image. +For example, if you have added a schema plugin `legacy_plugin` you should add +the following: + +```yaml +plugins: + options: + schemas/legacy_plugin: + python_package: legacy_plugin +``` + +Once the changes have been committed to the main branch, the new image will automatically +be generated. + +## FAQ/Trouble shooting + +_I get an_ `Error response from daemon: Head "https://ghcr.io/v2/blueraft/nomad-distribution-template/manifests/main": unauthorized` +_when trying to pull my docker image._ + +Most likely you have not made the package public or provided a personal access token (PAT). +You can read how to make your package public in the GitHub docs [here](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility) +or how to configure a PAT (if you want to keep the distribution private) in the GitHub +docs [here](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic). diff --git a/configs/nomad.yaml b/configs/nomad.yaml index b23f063..7950404 100644 --- a/configs/nomad.yaml +++ b/configs/nomad.yaml @@ -11,7 +11,7 @@ north: tools: options: jupyter: - image: ghcr.io/fairmat-nfdi/nomad-distribution-template/jupyter:main + image: ghcr.io/blueraft/nomad-distribution-template/jupyter:main meta: deployment: "oasis" diff --git a/docker-compose.yaml b/docker-compose.yaml index 2fbfb4d..7d48ea3 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -68,7 +68,7 @@ services: # nomad worker (processing) worker: restart: unless-stopped - image: ghcr.io/fairmat-nfdi/nomad-distribution-template:main + image: ghcr.io/blueraft/nomad-distribution-template:main container_name: nomad_oasis_worker environment: NOMAD_SERVICE: nomad_oasis_worker @@ -91,7 +91,7 @@ services: # nomad app (api + proxy) app: restart: unless-stopped - image: ghcr.io/fairmat-nfdi/nomad-distribution-template:main + image: ghcr.io/blueraft/nomad-distribution-template:main container_name: nomad_oasis_app environment: NOMAD_SERVICE: nomad_oasis_app @@ -130,7 +130,7 @@ services: # nomad remote tools hub (JupyterHUB, e.g. for AI Toolkit) north: restart: unless-stopped - image: ghcr.io/fairmat-nfdi/nomad-distribution-template:main + image: ghcr.io/blueraft/nomad-distribution-template:main container_name: nomad_oasis_north environment: NOMAD_SERVICE: nomad_oasis_north @@ -165,7 +165,7 @@ services: # to enable the logtransfer service run "docker compose --profile with_logtransfer up" logtransfer: restart: unless-stopped - image: ghcr.io/fairmat-nfdi/nomad-distribution-template:main + image: ghcr.io/blueraft/nomad-distribution-template:main container_name: nomad_oasis_logtransfer environment: NOMAD_SERVICE: nomad_oasis_logtransfer diff --git a/template_README.md b/template_README.md deleted file mode 100644 index 3bf2422..0000000 --- a/template_README.md +++ /dev/null @@ -1,164 +0,0 @@ -![docker image](https://github.com/GITHUB_REPOSITORY/actions/workflows/docker-publish.yml/badge.svg) - -# GITHUB_REPOSITORY_OWNER's NOMAD Oasis Distribution - -This is the NOMAD Oasis distribution of GITHUB_REPOSITORY_OWNER. -Below are instructions for how to [deploy this distribution](#deploying-the-distribution) -and how to customize it through [adding plugins](#adding-a-plugin). - -> [!IMPORTANT] -> Depending on the settings of the owner of this repository, the distributed image might -> be private and require authentication to pull. -> If you want to keep the image private you need to configure and use a personal access -> token (PAT) according to the instructions in the GitHub docs [here](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic). -> If you want to make the image public (recommended), you should make sure that your -> organization settings allow public packages and make this package public after building it. -> You can read more about this in the GitHub docs [here](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility). - -> [!TIP] -> In order for others to find and learn from your distribution we in FAIRmat would -> greatly appreciate it if you would add the topic `nomad-distribution` by clicking the -> ⚙️ next to "About" on the main GitHub page for this repository. - -## Deploying the distribution - -Below are instructions for how to deploy this NOMAD Oasis distribution -[for a new Oasis](#for-a-new-oasis) and [for an existing Oasis](#for-an-existing-oasis) - -### For a new Oasis - -1. Make sure you have [docker](https://docs.docker.com/engine/install/) installed. - Docker nowadays comes with `docker compose` built in. Prior, you needed to - install the stand-alone [docker-compose](https://docs.docker.com/compose/install/). - -2. Clone the repository or download the repository as a zip file. - -```sh -git clone https://github.com/GITHUB_REPOSITORY.git -cd GITHUB_REPOSITORY_NAME -``` - -or - -```sh -curl-L -o GITHUB_REPOSITORY_NAME.zip "https://github.com/GITHUB_REPOSITORY/archive/main.zip" -unzip GITHUB_REPOSITORY_NAME.zip -cd GITHUB_REPOSITORY_NAME -``` - -3. _On Linux only,_ recursively change the owner of the `.volumes` directory to the nomad user (1000) - -```sh -sudo chown -R 1000 .volumes -``` - -4. Pull the images specified in the `docker-compose.yaml` - -Note that the image needs to be public or you need to provide a PAT (see "Important" note above). - -```sh -docker compose pull -``` - -5. And run it with docker compose in detached (--detach or -d) mode - -```sh -docker compose up -d -``` - -6. Optionally you can now test that NOMAD is running with - -``` -curl localhost/nomad-oasis/alive -``` - -7. Finally, open [http://localhost/nomad-oasis](http://localhost/nomad-oasis) in your browser to start using your new NOMAD Oasis. - -Whenever you update your image you need to shut down NOMAD using - -```sh -docker compose down -``` - -and then repeat steps 4. and 5. above. - -#### NOMAD Remote Tools Hub (NORTH) - -To run NORTH (the NOMAD Remote Tools Hub), the `hub` container needs to run docker and -the container has to be run under the docker group. You need to replace the default group -id `991` in the `docker-compose.yaml`'s `hub` section with your systems docker group id. -Run `id` if you are a docker user, or `getent group | grep docker` to find your -systems docker gid. The user id 1000 is used as the nomad user inside all containers. - -You can find more details on setting up and maintaining an Oasis in the NOMAD docs here: -[nomad-lab.eu/prod/v1/docs/oasis/install.html](https://nomad-lab.eu/prod/v1/docs/oasis/install.html) - -### For an existing Oasis - -If you already have an Oasis running you only need to change the image being pulled in -your `docker-compose.yaml` with `ghcr.io/GITHUB_REPOSITORY:main` for the services -`worker`, `app`, `north`, and `logtransfer`. - -If you want to use the `nomad.yaml` from this repository you also need to comment out -the inclusion of the `nomad.yaml` under the volumes key of those services in the -`docker-compose.yaml`. - -```yaml -volumes: - # - ./configs/nomad.yaml:/app/nomad.yaml -``` - -To run the new image you can follow steps 5. and 6. [above](#for-a-new-oasis). - -## Adding a plugin - -To add a new plugin to the docker image you should add it to the plugins table in the `pyproject.toml` file. - -Here you can put either plugins distributed to PyPI, e.g. - -``` -nomad-material-processing -``` - -or plugins in a git repository with either the commit hash - -``` -"nomad-measurements @ git+https://github.com/FAIRmat-NFDI/nomad-measurements.git@71b7e8c9bb376ce9e8610aba9a20be0b5bce6775" -``` - -or with a tag - -``` -"nomad-measurements @ git+https://github.com/FAIRmat-NFDI/nomad-measurements.git@v0.0.4" -``` - -To add a plugin in a subdirectory of a git repository you can use the `subdirectory` option, e.g. - -``` -"ikz_pld_plugin @ git+https://github.com/FAIRmat-NFDI/AreaA-data_modeling_and_schemas.git@30fc90843428d1b36a1d222874803abae8b1cb42#subdirectory=PVD/PLD/jeremy_ikz/ikz_pld_plugin" -``` - -If the plugin is new, you also need to add it under `plugins` in the [nomad.yaml](nomad.yaml) -config file that will be included in the image. -For example, if you have added a schema plugin `legacy_plugin` you should add -the following: - -```yaml -plugins: - options: - schemas/legacy_plugin: - python_package: legacy_plugin -``` - -Once the changes have been committed to the main branch, the new image will automatically -be generated. - -## FAQ/Trouble shooting - -_I get an_ `Error response from daemon: Head "https://ghcr.io/v2/GITHUB_REPOSITORY/manifests/main": unauthorized` -_when trying to pull my docker image._ - -Most likely you have not made the package public or provided a personal access token (PAT). -You can read how to make your package public in the GitHub docs [here](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility) -or how to configure a PAT (if you want to keep the distribution private) in the GitHub -docs [here](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic).