feat: add support for Python poetry.lock files (#9)

* feat: add support for Python poetry.lock files * style: format Python files with black and isort * style: adhere to yamllint and format action.yml * style: adhere to markdownlint and format README * docs: update README for 1.4.1 release * refactor: update action.yml dependencies and format * build: Ensure each dependent action is pinned to the latest major version tag * style: correct for basic Python linting issues * style: format and cleanup analyze.py * style: provide descriptive step names in action.yml
phylum-dev · Apr 13, 2022 · 6e4ad7d · 6e4ad7d
1 parent a9b3399
commit 6e4ad7d
Show file tree

Hide file tree

Showing 5 changed files with 331 additions and 271 deletions.
diff --git a/README.md b/README.md
@@ -1,18 +1,26 @@
 # phylum-analyze-pr-action
 A GitHub Action to automatically analyze Pull Requests for changes to package manager lockfiles using Phylum.
 
-Phylum provides a complete risk analyis of "open-source packages" (read: untrusted software from random Internet strangers). Phylum evolved forward from legacy SCA tools to defend from supply-chain malware, malicious open-source authors, and engineering risk, in addtion to software vulnerabilities and license risks. To learn more, please see [our website](https://phylum.io)
+Phylum provides a complete risk analyis of "open-source packages" (read: untrusted software from random Internet
+strangers). Phylum evolved forward from legacy SCA tools to defend from supply-chain malware, malicious open-source
+authors, and engineering risk, in addtion to software vulnerabilities and license risks. To learn more, please see
+[our website](https://phylum.io)
 
-This action enables users to configure thresholds for each of Phylum's five risk domain scores. If a package risk domain score is below the threshold, the action will fail the check on the pull request. When packages fail the risk analysis, a comment is created on the PR to summarize the issues.
+This action enables users to configure thresholds for each of Phylum's five risk domain scores. If a package risk
+domain score is below the threshold, the action will fail the check on the pull request. When packages fail the risk
+analysis, a comment is created on the PR to summarize the issues.
 
 ## Features
 - configurable risk domain thresholds
-- uses [peter-evans/create-or-update-comment](https://github.com/marketplace/actions/create-or-update-comment) to add comments to PRs
+- uses [peter-evans/create-or-update-comment](https://github.com/marketplace/actions/create-or-update-comment)
+  to add comments to PRs
 
 ## Getting Started
-1. Create a workflow in a repository that uses the workflow definition listed below as an example. 
-2. Be sure to include the base branches you use for development, the defaults are set to `master` and `main`. 
-3. Define risk domain thresholds using `vul_threshold`, `mal_threshold`, etc to define a score requirement. A Phylum project score requirement of 60 is defined as `0.6`, for example. 
+1. Create a workflow in a repository that uses the workflow definition listed below as an example
+2. Be sure to include the base/default branches used for development, where the defaults are set to `master` and `main`
+3. Define risk domain thresholds using `vul_threshold`, `mal_threshold`, etc. to define a score requirement
+   1. For example, a Phylum project score requirement of 60 is defined as `0.6`
+4. Additional inputs can be used - see [action.yml](action.yml) for full list
 
 ```yaml
 on:
@@ -26,9 +34,9 @@ jobs:
     runs-on: ubuntu-latest
     name: A job to analyze PR with phylum
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - id: analyze-pr-test
-        uses: phylum-dev/phylum-analyze-pr-action@v1.4
+        uses: phylum-dev/phylum-analyze-pr-action@v1
         with:
           vul_threshold: 0.6
           mal_threshold: 0.6
@@ -39,34 +47,39 @@ jobs:
 ```
 
 ### Supported lockfiles
-- requirements.txt (Python PyPI)
-- package-lock.json (JavaScript/TypeScript NPM)
-- yarn.lock (JavaScript/TypeScript NPM)
-- Gemfile.lock (Ruby Rubygems/Bundler)
+- `requirements.txt` (Python PyPI)
+- `poetry.lock` (Python PyPI)
+- `package-lock.json` (JavaScript/TypeScript NPM)
+- `yarn.lock` (JavaScript/TypeScript NPM)
+- `Gemfile.lock` (Ruby Rubygems/Bundler)
 
 ### Requirements
 - active Phylum account ([Register here](https://app.phylum.io/auth/registration))
-- GitHub repository secret defined: PHYLUM_TOKEN (extracted from Phylum CLI configuration file "offline_access")
-  1. Ensure you've updated the Phylum CLI on a local installation to a version >= `1.2.0`
-  2. Successfully authenticate using Phylum CLI. This will ensure the token is populated in the phylum config file `~/.phylum/settings.yaml` in stanza `offline_access`
-  3. Copy the token value in the `offline_access` stanza
-  4. Create a new GitHub secret in the desired repository. This can be done through the GitHub web UI or using the gh command line tool: `gh secret set PHYLUM_TOKEN -b <token_value>`
-  5. Optionally, you can remove the vestigial `PHYLUM_USER` and `PHYLUM_PASS` GitHub secrets as they are no longer used.
-- concrete package versions (only applicable for requirements.txt)
+- GitHub repository secret defined: `PHYLUM_TOKEN`
+  1. Ensure you've updated the Phylum CLI on a local installation to a version >= `2.0.1`
+  2. Successfully authenticate using Phylum CLI to ensure the token is populated and correct
+  3. Copy the token value from the output of the `phylum auth token` command
+  4. Create a new GitHub secret named `PHYLUM_TOKEN` in the desired repository, through the GitHub web UI or using the gh command line tool: `gh secret set PHYLUM_TOKEN -b <token_value>`
+- concrete package versions (only applicable for `requirements.txt`)
 - existing Phylum project for repository (`.phylum_project` must be present)
 
 ### Known Issues
-~~1. Incomplete packages: if Phylum hasn't yet analyzed a package requested by this action, the action will fail with an exit code of 5. This is momentarily preferable than waiting.~~
+- [Issue tracker](https://github.com/phylum-dev/phylum-analyze-pr-action/issues)
+- [Open bugs](https://github.com/phylum-dev/phylum-analyze-pr-action/labels/%F0%9F%95%B7%EF%B8%8F%20bug)
 
 ### Incomplete Packages
-Sometimes, users will request risk analysis information for open-source packages Phylum has not yet processed. When this occurs, Phylum cannot reasonably provide risk scoring information until those packages have been processed. 
+Sometimes, users will request risk analysis information for open-source packages Phylum has not yet processed.
+When this occurs, Phylum cannot reasonably provide risk scoring information until those packages have been processed.
 
-New in `v1.4`, `phylum-analyze-pr-action` will:
+Starting with `v1.4.0`, `phylum-analyze-pr-action` will:
 1. Detect the case of incomplete packages
-2. Return an exit code of 0 (a "passing" mark in GitHub Action parlance). This is to avoid failing a check in the PR with incomplete information.
-3. Add a comment to the PR indicating that there were incomplete packages. The comment will advise users to wait 30m and re-run the check on the Pull Request. This will give Phylum sufficient time to download, process and analyze the incomplete packages.
-4. When the check is run a second time, another comment will be added to the Pull Request noting the result of the risk analysis operation. 
+2. Return an exit code of 0 (a "passing" mark in GitHub Action parlance)
+   1. This is to avoid failing a check in the PR with incomplete information
+3. Add a comment to the PR indicating that there were incomplete packages
+   1. The comment will advise users to wait 30m and re-run the check on the Pull Request
+   2. This will give Phylum sufficient time to download, process and analyze the incomplete packages
+4. When the check is run a second time, another comment will be added to the Pull Request noting the result of the
+   risk analysis operation.
 
 ### Example comment
 ![image](https://user-images.githubusercontent.com/132468/140830714-24acc278-0102-4613-b006-6032a62b6896.png)
-
diff --git a/action.yml b/action.yml
@@ -1,9 +1,10 @@
-#                   _
-#  __ _ _ __   __ _| |_   _ _______
-# / _` | '_ \ / _` | | | | |_  / _ \
-#| (_| | | | | (_| | | |_| |/ /  __/
-# \__,_|_| |_|\__,_|_|\__, /___\___|
-#                    |___/
+#                    _
+#   __ _ _ __   __ _| |_   _ _______
+#  / _` | '_ \ / _` | | | | |_  / _ \
+# | (_| | | | | (_| | | |_| |/ /  __/
+#  \__,_|_| |_|\__,_|_|\__, /___\___|
+#                     |___/
+---
 name: 'Analyze PR'
 description: 'Analyze Pull request'
 inputs:
@@ -42,27 +43,27 @@ inputs:
 runs:
   using: "composite"
   steps:
-    - id: phylum-test
-      uses: phylum-dev/install-phylum-latest-action@v1.3
+    - name: Install phylum CLI
+      uses: phylum-dev/install-phylum-latest-action@v1
       with:
         phylum_token: ${{ inputs.phylum_token }}
         phylum_version: ${{ inputs.phylum_version }}
 
-    - name: Check for previous comment
+    - name: Check for INCOMPLETE comment
       uses: peter-evans/find-comment@v1
       id: fc
       with:
         issue-number: ${{ github.event.pull_request.number }}
         body-includes: INCOMPLETE
 
-    - name: Store result of id=fc in environment
+    - name: Record presence of Phylum INCOMPLETE comment
       shell: bash
       if: "contains(steps.fc.outputs.comment-body, 'Phylum')"
       run: |
         echo "storing PREVIOUS_INCOMPLETE"
         echo PREVIOUS_INCOMPLETE=1 >> $GITHUB_ENV
 
-    - name: Check for existing project
+    - name: Check for existing .phylum_project
       shell: bash
       run: |
         result=$(find . -maxdepth 1 -iname ".phylum_project")
@@ -73,29 +74,31 @@ runs:
           exit 11
         fi
 
-    - name: Generate Phylum label
+    - name: Generate PHYLUM_LABEL
       shell: bash
-      run: |
-        echo PHYLUM_LABEL="GHA-PR${{ github.event.number }}-${GITHUB_HEAD_REF}" >> $GITHUB_ENV
+      run: echo PHYLUM_LABEL="GHA-PR${{ github.event.number }}-${GITHUB_HEAD_REF}" >> $GITHUB_ENV
 
-    - uses: actions/setup-python@v2
+    - name: Setup Python
+      uses: actions/setup-python@v3
       with:
         python-version: '3.x'
 
-    - name: install python dependencies
+    - name: Install python dependencies
       shell: bash
       run: |
-        pip install unidiff
+        python -m pip install -U pip setuptools
+        python -m pip install unidiff packaging
 
-    - name: run analyze.py pr_type
+    - name: Determine the PR type
       shell: bash
       run: python $GITHUB_ACTION_PATH/analyze.py "pr_type" $GITHUB_REPOSITORY ${{ github.event.number }}
 
-    - name: cat prtype
+    - name: Display the PR type
       shell: bash
       run: cat ~/prtype.txt
 
-    - id: get-prtype
+    - name: Make PR type available for future steps
+      id: get-prtype
       shell: bash
       run: |
         ret="$(cat ~/prtype.txt)"
@@ -104,42 +107,45 @@ runs:
         ret="${ret//$'\r'/'%0A'}"
         echo "::set-output name=prtype::$ret"
 
-    - id: should-proceed
+    - name: Bail when no updates to analyze
+      id: should-proceed
       shell: bash
       if: "contains(steps.get-prtype.outputs.prtype, 'NA')"
       run: |
         echo 'exiting with 0 since package dependency files were not modified'
         echo '0' > $HOME/returncode.txt
 
-    - name: Analyze project lockfile
+    - name: Analyze project lockfile with phylum CLI
       shell: bash
       if: "!contains(steps.get-prtype.outputs.prtype, 'NA')"
       run: |
         export PATH="$HOME/.phylum:$PATH"
         pushd $GITHUB_WORKSPACE || exit 11
-          phylum analyze -l $PHYLUM_LABEL ${{ steps.get-prtype.outputs.prtype }} --verbose --json > ~/phylum_analysis.json
-          echo "[*] Analyzed ${{ steps.get-prtype.outputs.prtype }} under label ${PHYLUM_LABEL} and wrote results to ~/phylum_analysis.json"
+        phylum analyze -l $PHYLUM_LABEL ${{ steps.get-prtype.outputs.prtype }} --verbose --json > ~/phylum_analysis.json
+        echo "[*] Analyzed ${{ steps.get-prtype.outputs.prtype }} under label ${PHYLUM_LABEL} and wrote results to ~/phylum_analysis.json"
         popd
 
-
-    # - name: tmate
-      # uses: mxschmitt/action-tmate@v3
-
-    - name: invoke test matrix
+    - name: Invoke test matrix
       shell: bash
       if: "contains(inputs.invoke_test_matrix, 'true')"
-      run: |
-        python $GITHUB_ACTION_PATH/test_matrix.py
+      run: python $GITHUB_ACTION_PATH/test_matrix.py
 
-    - name: python script analyze.py
+    - name: Compare added dependencies in PR to analysis results
       shell: bash
       if: "!contains(steps.get-prtype.outputs.prtype, 'NA')"
-      run: python $GITHUB_ACTION_PATH/analyze.py "analyze" $GITHUB_REPOSITORY ${{ github.event.number }} ${{ inputs.vul_threshold }} ${{ inputs.mal_threshold }} ${{ inputs.eng_threshold }} ${{ inputs.lic_threshold }} ${{ inputs.aut_threshold }}
-
-    # - name: tmate
-      # uses: mxschmitt/action-tmate@v3
-
-    - id: get-returncode
+      run: >
+        python $GITHUB_ACTION_PATH/analyze.py
+        "analyze"
+        $GITHUB_REPOSITORY
+        ${{ github.event.number }}
+        ${{ inputs.vul_threshold }}
+        ${{ inputs.mal_threshold }}
+        ${{ inputs.eng_threshold }}
+        ${{ inputs.lic_threshold }}
+        ${{ inputs.aut_threshold }}
+
+    - name: Get return code
+      id: get-returncode
       shell: bash
       run: |
         ret="$(cat ~/returncode.txt)"
@@ -149,14 +155,15 @@ runs:
         echo "::set-output name=ret::$ret"
 
     # This will catch SUCCESS cases
-    - name: return 0 for success
+    - name: Return 0 for success
       shell: bash
       if: "contains(steps.get-returncode.outputs.ret, '0')"
       run: |
-        echo 'exiting with 0 for success'
+        echo "exiting with 0 for success"
         exit 0
 
-    - id: get-comment-body
+    - name: Get comment body
+      id: get-comment-body
       # this will have to check for 1 or 5 AND if on the second run
       # if: "contains(steps.get-returncode.outputs.ret, '1')"
       if: "steps.get-returncode.outputs.ret > 0"
@@ -171,26 +178,25 @@ runs:
     - name: Set comment
       # This will have to check for 1 or 5
       # Could check for > 0 ?
-      #if: "contains(steps.get-returncode.outputs.ret, '1')"
+      # if: "contains(steps.get-returncode.outputs.ret, '1')"
       if: "steps.get-returncode.outputs.ret > 0"
-      uses: peter-evans/create-or-update-comment@v1
+      uses: peter-evans/create-or-update-comment@v2
       with:
         issue-number: ${{ github.event.pull_request.number }}
         body: ${{ steps.get-comment-body.outputs.body }}
 
     # This will catch INCOMPLETE and COMPLETE_SUCCESS
-    - name: handle ret values of 4 or 5
+    - name: Handle return values of 4 or 5
       shell: bash
       if: "steps.get-returncode.outputs.ret >= 4"
       run: |
         echo 'exiting with 0 for success - ret = ${{ steps.get-returncode.outputs.ret }}'
         exit 0
 
     # This will catch FAILURE and COMPLETE_FAILURE
-    - name: return 1 for risk analysis failure
+    - name: Handle risk analysis failures
       shell: bash
       if: "contains(steps.get-returncode.outputs.ret, '1')"
       run: |
-        echo 'exiting with 1 for risk analysis failure'
+        echo "exiting with 1 for risk analysis failure"
         exit 1
-