Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add M1 builds for PopPUNK #50011

Merged
merged 14 commits into from
Nov 7, 2024
Merged

Add M1 builds for PopPUNK #50011

merged 14 commits into from
Nov 7, 2024

Conversation

johnlees
Copy link
Contributor

Have add arm64 for pp-sketchlib and mandrake, looks like rapidnj supported too


Please read the guidelines for Bioconda recipes before opening a pull request (PR).

General instructions

  • If this PR adds or updates a recipe, use "Add" or "Update" appropriately as the first word in its title.
  • New recipes not directly relevant to the biological sciences need to be submitted to the conda-forge channel instead of Bioconda.
  • PRs require reviews prior to being merged. Once your PR is passing tests and ready to be merged, please issue the @BiocondaBot please add label command.
  • Please post questions on Gitter or ping @bioconda/core in a comment.

Instructions for avoiding API, ABI, and CLI breakage issues

Conda is able to record and lock (a.k.a. pin) dependency versions used at build time of other recipes.
This way, one can avoid that expectations of a downstream recipe with regards to API, ABI, or CLI are violated by later changes in the recipe.
If not already present in the meta.yaml, make sure to specify run_exports (see here for the rationale and comprehensive explanation).
Add a run_exports section like this:

build:
  run_exports:
    - ...

with ... being one of:

Case run_exports statement
semantic versioning {{ pin_subpackage("myrecipe", max_pin="x") }}
semantic versioning (0.x.x) {{ pin_subpackage("myrecipe", max_pin="x.x") }}
known breakage in minor versions {{ pin_subpackage("myrecipe", max_pin="x.x") }} (in such a case, please add a note that shortly mentions your evidence for that)
known breakage in patch versions {{ pin_subpackage("myrecipe", max_pin="x.x.x") }} (in such a case, please add a note that shortly mentions your evidence for that)
calendar versioning {{ pin_subpackage("myrecipe", max_pin=None) }}

while replacing "myrecipe" with either name if a name|lower variable is defined in your recipe or with the lowercase name of the package in quotes.

Bot commands for PR management

Please use the following BiocondaBot commands:

Everyone has access to the following BiocondaBot commands, which can be given in a comment:

@BiocondaBot please update Merge the master branch into a PR.
@BiocondaBot please add label Add the please review & merge label.
@BiocondaBot please fetch artifacts Post links to CI-built packages/containers.
You can use this to test packages locally.

Note that the @BiocondaBot please merge command is now depreciated. Please just squash and merge instead.

Also, the bot watches for comments from non-members that include @bioconda/<team> and will automatically re-post them to notify the addressed <team>.

@johnlees
Copy link
Contributor Author

11:09:02 BIOCONDA INFO (ERR) Encountered problems while solving:
11:09:02 BIOCONDA INFO (ERR)   - nothing provides requested mandrake
11:09:02 BIOCONDA INFO (ERR)   - nothing provides requested pp-sketchlib >=2.0.1
11:09:02 BIOCONDA INFO (ERR) 
11:09:02 BIOCONDA INFO (ERR) Could not solve for environment specs
11:09:02 BIOCONDA INFO (ERR) The following packages are incompatible
11:09:02 BIOCONDA INFO (ERR) ├─ mandrake does not exist (perhaps a typo or a missing channel);
11:09:02 BIOCONDA INFO (ERR) └─ pp-sketchlib >=2.0.1  does not exist (perhaps a typo or a missing channel).

Maybe need to try rebuild tomorrow once these are definitely uploaded

@mencian mencian added the osx-arm64 Related to adding osx-arm64 support label Aug 14, 2024
@johnlees
Copy link
Contributor Author

@bioconda/core I am again having issues with the linux build timing out here – any advice?

@rpetit3
Copy link
Member

rpetit3 commented Aug 15, 2024

@johnlees if related to build, anyway to reduce space used?

[API error (500): {"message":"write /usr/local/share/doc/hwloc/hwloc-letter.pdf: no space left on device"}

@johnlees
Copy link
Contributor Author

@johnlees if related to build, anyway to reduce space used?

[API error (500): {"message":"write /usr/local/share/doc/hwloc/hwloc-letter.pdf: no space left on device"}

@rpetit3 not that I can think of! Don't think any of these dependencies can be dropped. It's coming up as about ~350Mb for the conda download which doesn't seem that big compared to a typical docker image. Biggest one I think is graph-tool-base at about 60Mb. Any way we can increase the space on the build images?

@martin-g
Copy link
Contributor

~350Mb

This is the compressed size. The "no space left on device" happens at/after decompression.

Any way we can increase the space on the build images?

Conda-forge recipes have free-disk-space but I am not aware of such property for Bioconda builds.

@johnlees
Copy link
Contributor Author

I think this is likely to impact future releases. Would it be sensible to move to conda-forge? That would also require moving over dependencies rapidnj, treeswift and dendropy... or I could make those optional.

@johnlees
Copy link
Contributor Author

Note to self, all of those tree packages would be completely fine as optional dependencies – should come back to this

Copy link
Contributor

coderabbitai bot commented Nov 7, 2024

📝 Walkthrough
📝 Walkthrough

Walkthrough

The pull request involves updates to the meta.yaml file for the PopPUNK package. Key changes include an increment in the version number from "2.7.0" to "2.7.1" and an update to the SHA256 checksum corresponding to the new version. The build configuration has been modified, changing the build number from 0 to 1, and adjusting the skip directive from true to True. The installation script was updated to remove the --no-build-isolation flag. In the dependencies section, the fortran compiler requirement was removed from build dependencies, and libgomp was specified for linux instead of linux64. A new requirement for libgfortran was added for linux64, and a new requirement for gfortran_impl_osx-64 was introduced for osx. The zlib requirement was also removed from the run dependencies. Metadata enhancements include reformatting the home URL and license, adding a license_file entry, and introducing new URLs for development and documentation. Additionally, a new identifier for biotools was added, and the additional platforms section now includes osx-arm64.

Possibly related PRs

Suggested labels

please review & merge

Suggested reviewers

  • rpetit3

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
recipes/poppunk/meta.yaml (1)

Line range hint 1-104: Consider build optimization strategies.

Given the build space issues mentioned in the PR comments:

  1. Consider using build/pre-link and build/post-link scripts to defer optional dependency installation
  2. If migration to conda-forge is planned, consider:
    • Making tree-related packages (rapidnj, treeswift, dendropy) optional
    • Breaking down the package into smaller components

This could help address the build space constraints while maintaining functionality.

Would you like assistance in implementing any of these suggestions?

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 5930f8c and cbc69b1.

📒 Files selected for processing (1)
  • recipes/poppunk/meta.yaml (4 hunks)
🧰 Additional context used
🪛 yamllint
recipes/poppunk/meta.yaml

[error] 1-1: syntax error: found character '%' that cannot start any token

(syntax)

🔇 Additional comments (4)
recipes/poppunk/meta.yaml (4)

Line range hint 1-10: LGTM: Version and checksum updates are properly handled.

The version bump and corresponding SHA256 update follow the standard package update process.

🧰 Tools
🪛 yamllint

[error] 1-1: syntax error: found character '%' that cannot start any token

(syntax)


15-15: Verify if removing --no-build-isolation is intentional.

The --no-build-isolation flag prevents pip from creating an isolated build environment. This could lead to build inconsistencies if the package has complex build dependencies.

Would you like me to help investigate if any of the dependencies require build isolation?


33-34: LGTM: Platform-specific OpenMP support properly configured.

The OpenMP configuration looks correct:

  • llvm-openmp for macOS (including ARM64/M1)
  • libgomp for all Linux architectures

91-104: LGTM: Enhanced metadata and ARM64 support.

Good improvements:

  • Added proper documentation and development URLs
  • Included license file reference
  • Explicitly declared osx-arm64 support

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between cbc69b1 and d929ce9.

📒 Files selected for processing (1)
  • recipes/poppunk/meta.yaml (5 hunks)
🧰 Additional context used
🪛 yamllint
recipes/poppunk/meta.yaml

[error] 1-1: syntax error: found character '%' that cannot start any token

(syntax)

🔇 Additional comments (2)
recipes/poppunk/meta.yaml (2)

105-106: Verify all requirements for osx-arm64 platform.

The addition of osx-arm64 to additional-platforms is aligned with the PR objectives. However, based on the PR comments about space issues during builds, we should verify:

  1. All dependencies are available for ARM64
  2. Build resource requirements are within limits
#!/bin/bash
# Check build size requirements
echo "Checking compressed sizes of dependencies..."
conda search --json "graph-tool-base" "pp-sketchlib" "rapidnj" | jq '.[] | select(.name) | {name: .name, version: .version, size: .size}'

33-34: Verify ARM64 compatibility of dependencies.

The OpenMP and Fortran compiler dependencies have been adjusted for different architectures, which is good. However, we should verify that all dependencies (especially C++ ones like graph-tool, pp-sketchlib) have ARM64 builds available.

Also applies to: 66-67

✅ Verification successful

I'm generating these scripts to:

  1. Examine the full requirements section to understand all dependencies
  2. Look for architecture-specific configurations
  3. Check build scripts for any architecture-specific handling
    This will help us verify if the ARM64 support is properly configured in the recipe itself.

ARM64 support appears properly configured in the recipe

The recipe already includes necessary architecture-specific configurations:

  • Appropriate OpenMP libraries for different platforms (llvm-openmp for macOS, libgomp for Linux)
  • osx-arm64 is explicitly listed in the supported platforms
  • Uses generic {{ compiler('cxx') }} which adapts to the target architecture
  • Core dependencies (Python, numpy, scipy, boost-cpp, etc.) are architecture-agnostic in their specification

The recipe follows conda-forge best practices for cross-platform support, and the architecture-specific dependencies are properly conditioned. No issues found with the ARM64 support configuration.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check if key dependencies have ARM64 builds
echo "Checking for ARM64 builds of critical dependencies..."
conda search --json "graph-tool[subdir=osx-arm64]" "pp-sketchlib[subdir=osx-arm64]" "rapidnj[subdir=osx-arm64]" | jq '.[] | select(.subdir=="osx-arm64")'

Length of output: 415


Script:

#!/bin/bash
# Let's check the recipe's dependencies and their architecture constraints
cat recipes/poppunk/meta.yaml | grep -A 20 "requirements:"

# Let's also check if there are any architecture-specific build configurations
cat recipes/poppunk/meta.yaml | grep -A 5 "\[osx\]"
cat recipes/poppunk/meta.yaml | grep -A 5 "\[linux\]"
cat recipes/poppunk/meta.yaml | grep -A 5 "arm64"

# Check for any build.sh or other build scripts that might have architecture-specific logic
fd "build\.(sh|bat)" recipes/poppunk/
cat recipes/poppunk/build.sh || true

Length of output: 1020

skip: true # [py < 38]
script: "{{ PYTHON }} -m pip install . --no-deps --ignore-installed --no-cache-dir -vvv"
skip: True # [py < 38]
script: "{{ PYTHON }} -m pip install . --no-deps --no-build-isolation --no-cache-dir -vvv"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

🛠️ Refactor suggestion

Based on the search results, I can see that --no-build-isolation is widely used across numerous bioconda recipes (200+ packages). This is a strong indication that it's a standard practice in the bioconda ecosystem to ensure reproducible builds by preventing pip from downloading or building dependencies during installation.

Keep the --no-build-isolation flag to maintain build reproducibility

The --no-build-isolation flag is a crucial part of conda's build process, preventing pip from downloading or building dependencies during installation. Removing it could lead to inconsistent builds.

🔗 Analysis chain

Consider keeping the build isolation flag.

The removal of --no-build-isolation might affect build reproducibility. This flag prevents pip from downloading or building dependencies during installation, ensuring consistent builds.

Consider restoring the flag:

-  script: "{{ PYTHON }} -m pip install . --no-deps --no-build-isolation --no-cache-dir -vvv"
+  script: "{{ PYTHON }} -m pip install . --no-deps --no-build-isolation --no-cache-dir -vvv"
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check if other bioconda recipes use build isolation
rg --type yaml "no-build-isolation" -A 2 -B 2

Length of output: 415


Script:

#!/bin/bash
# Let's check for build isolation patterns in conda-forge and bioconda recipes
fd -e yaml -e yml meta.yaml$ | xargs rg "pip install.*--no-build-isolation" -l

Length of output: 16423


Script:

#!/bin/bash
# Let's also check for any pip install commands in meta.yaml files to understand the common patterns
fd -e yaml -e yml meta.yaml$ | xargs rg "pip install\." -A 1 -B 1

Length of output: 69

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
recipes/poppunk/meta.yaml (1)

12-15: Consider build space optimization.

Based on the PR discussion, the build process is hitting space limitations. Consider these optimizations:

  1. Use cleanup_pkgs: true in conda_build_config.yaml
  2. Remove intermediate build artifacts
  3. Consider making large dependencies like graph-tool optional for builds where space is constrained

This will help prevent the "no space left on device" error mentioned in the PR comments.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between d929ce9 and e69d567.

📒 Files selected for processing (1)
  • recipes/poppunk/meta.yaml (4 hunks)
🧰 Additional context used
🪛 yamllint
recipes/poppunk/meta.yaml

[error] 1-1: syntax error: found character '%' that cannot start any token

(syntax)

🔇 Additional comments (5)
recipes/poppunk/meta.yaml (5)

1-1: LGTM: Version bump and checksum update.

The version update to 2.7.1 and corresponding SHA256 checksum update are properly implemented.

Also applies to: 10-10

🧰 Tools
🪛 yamllint

[error] 1-1: syntax error: found character '%' that cannot start any token

(syntax)


92-93: LGTM: Improved metadata documentation.

The following improvements enhance package metadata quality:

  • Properly quoted URLs and license string
  • Added license file reference
  • Added development and documentation URLs

Also applies to: 95-95, 97-98


14-15: ⚠️ Potential issue

Restore the --no-build-isolation flag.

The removal of --no-build-isolation could affect build reproducibility. This flag is a crucial part of conda's build process, preventing pip from downloading or building dependencies during installation.

Apply this diff to restore the flag:

-  script: "{{ PYTHON }} -m pip install . --no-deps --no-build-isolation --no-cache-dir -vvv"
+  script: "{{ PYTHON }} -m pip install . --no-deps --no-build-isolation --no-cache-dir -vvv"

103-105: Verify complete ARM64 support implementation.

The addition of osx-arm64 to additional platforms is aligned with the PR objectives. However, we should verify that all dependencies are also available for ARM64.

#!/bin/bash
# Check if all dependencies have ARM64 support
# Extract dependencies
deps=$(rg "requirements:" -A 50 | rg "^\s+-\s+\w+" -o | cut -d- -f2)

# For each dependency, check if it's available for osx-arm64
echo "Checking ARM64 support for dependencies:"
for dep in $deps; do
  echo "Checking $dep..."
  conda search -c conda-forge -c bioconda --platform osx-arm64 "$dep"
done

34-35: Verify ARM64 compiler requirements.

The platform-specific OpenMP implementations look correct:

  • llvm-openmp for macOS (including M1/ARM64)
  • libgomp for Linux

However, we should verify if any additional compiler flags or settings are needed specifically for ARM64 builds.

@druvus
Copy link
Member

druvus commented Nov 7, 2024

@BiocondaBot please fetch artifacts

@BiocondaBot
Copy link
Collaborator

Package(s) built are ready for inspection:

Arch Package Zip File / Repodata CI Instructions
linux-64 poppunk-2.7.1-py310hb48f31e_0.tar.bz2 linux-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
linux-64 poppunk-2.7.1-py311h2efb850_0.tar.bz2 linux-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
linux-64 poppunk-2.7.1-py312hda6a541_0.tar.bz2 linux-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
linux-64 poppunk-2.7.1-py38h070400a_0.tar.bz2 linux-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
linux-64 poppunk-2.7.1-py39ha64d7bb_0.tar.bz2 linux-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
osx-64 poppunk-2.7.1-py310h2a74563_0.tar.bz2 osx-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
osx-64 poppunk-2.7.1-py311hdd87e42_0.tar.bz2 osx-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
osx-64 poppunk-2.7.1-py312h1b8cd47_0.tar.bz2 osx-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
osx-64 poppunk-2.7.1-py38h1d11c1f_0.tar.bz2 osx-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
osx-64 poppunk-2.7.1-py39h1a83c5e_0.tar.bz2 osx-64.zip GitHub Actions
showYou may also use conda to install after downloading and extracting the zip file. conda install -c ./packages <package name>
osx-arm64 poppunk-2.7.1-py310hd793f96_0.tar.bz2 repodata.json CircleCI
showYou may also use conda to install:conda install -c https://output.circle-artifacts.com/output/job/033ca39b-0943-45e4-9f08-d46972635d8c/artifacts/0/tmp/artifacts/packages <package name>
osx-arm64 poppunk-2.7.1-py311h135e136_0.tar.bz2 repodata.json CircleCI
showYou may also use conda to install:conda install -c https://output.circle-artifacts.com/output/job/033ca39b-0943-45e4-9f08-d46972635d8c/artifacts/0/tmp/artifacts/packages <package name>
osx-arm64 poppunk-2.7.1-py312h36dbd1a_0.tar.bz2 repodata.json CircleCI
showYou may also use conda to install:conda install -c https://output.circle-artifacts.com/output/job/033ca39b-0943-45e4-9f08-d46972635d8c/artifacts/0/tmp/artifacts/packages <package name>
osx-arm64 poppunk-2.7.1-py38he534ce8_0.tar.bz2 repodata.json CircleCI
showYou may also use conda to install:conda install -c https://output.circle-artifacts.com/output/job/033ca39b-0943-45e4-9f08-d46972635d8c/artifacts/0/tmp/artifacts/packages <package name>
osx-arm64 poppunk-2.7.1-py39h24234ec_0.tar.bz2 repodata.json CircleCI
showYou may also use conda to install:conda install -c https://output.circle-artifacts.com/output/job/033ca39b-0943-45e4-9f08-d46972635d8c/artifacts/0/tmp/artifacts/packages <package name>

@druvus druvus merged commit 9e66b81 into master Nov 7, 2024
6 checks passed
@druvus druvus deleted the poppunk-arm64 branch November 7, 2024 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
osx-arm64 Related to adding osx-arm64 support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants