Troubleshooting an environment that won't build

[talmo]

First off, amazing work on this whole project stack -- seriously a game-changer for leveraging WASM and getting around the issues with Pyodide!

I was hoping to give this template a spin as an entry point to making custom JupyterLite/Xeus environments with our package preinstalled.

Unfortunately, even though our package is on conda-forge and pure Python, it's been a bit painful to figure out which of our deps is failing the build.

Issue investigation

I'm trying to get a JupyterLite build with sleap-io. It's on conda-forge as a noarch package. The deps are minimal but here's the relevant bit from meta.yaml:

  run:
    - python >=3.7
    - ffmpeg
    - imageio
    - imageio-ffmpeg >=0.5.0
    - attrs
    - pandas
    - simplejson
    - h5py >=3.8.0
    - hdmf
    - numpy
    - opencv
    - pynwb
    - ndx-pose <0.2.0

The only change I made in my fork of the xeus-python-demo template was to add the sleap-io package in environment.yml:

name: xeus-python-kernel
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - python=3.11
  - xeus-python
  - ipycanvas
  - conda-forge::sleap-io

On the micromamba-based runner in the preconfigured GH Actions, I'm getting this error when trying to build it (with jupyter lite build --contents content --output-dir dist):

error    libmamba Could not solve for environment specs
    The following packages are incompatible
    └─ sleap-io is not installable because there are no viable options
       ├─ sleap-io [0.1.6|0.1.7|0.1.8] would require
       │  └─ av, which does not exist (perhaps a missing channel);
       └─ sleap-io 0.1.9 would require
          └─ ffmpeg, which does not exist (perhaps a missing channel).
critical libmamba Could not solve for environment specs
[LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus] [ERR] Command '['/home/runner/micromamba-bin/micromamba', 'create', '--yes', '--no-pyc', '--root-prefix', PosixPath('/tmp/tmpkfxuj4ha/env'), '--name', 'xeus-python-kernel', '--platform=emscripten-wasm[32](https://github.com/talmo/sleap-io-notebook/actions/runs/11089260612/job/30810170200#step:5:33)', '-c', 'https://repo.mamba.pm/emscripten-forge', '-c', 'conda-forge', 'python=3.11', 'xeus-python', 'ipycanvas', 'conda-forge::sleap-io', 'emscripten-abi=3.1.45']' returned non-zero exit status 1.

Full conda list of build-env

/home/runner/micromamba-bin/micromamba list -r /home/runner/micromamba -n build-env
  List of packages in environment: "/home/runner/micromamba/envs/build-env"
  
    Name                           Version         Build               Channel    
  ──────────────────────────────────────────────────────────────────────────────────
    _libgcc_mutex                  0.1             conda_forge         conda-forge
    _openmp_mutex                  4.5             2_gnu               conda-forge
    anyio                          4.6.0           pyhd8ed1ab_1        conda-forge
    appdirs                        1.4.4           pyh9f0ad1d_0        conda-forge
    argon2-cffi                    23.1.0          pyhd8ed1ab_0        conda-forge
    argon2-cffi-bindings           21.2.0          py312h66e93f0_5     conda-forge
    arrow                          1.3.0           pyhd8ed1ab_0        conda-forge
    attrs                          24.2.0          pyh71513ae_0        conda-forge
    beautifulsoup4                 4.12.3          pyha770c72_0        conda-forge
    bleach                         6.1.0           pyhd8ed1ab_0        conda-forge
    brotli-python                  1.1.0           py312h2ec8cdc_2     conda-forge
    bzip2                          1.0.8           h4bc722e_7          conda-forge
    ca-certificates                2024.8.30       hbcca054_0          conda-forge
    cached-property                1.5.2           hd8ed1ab_1          conda-forge
    cached_property                1.5.2           pyha770c72_1        conda-forge
    certifi                        2024.8.30       pyhd8ed1ab_0        conda-forge
    cffi                           1.17.1          py312h06ac9bb_0     conda-forge
    charset-normalizer             3.3.2           pyhd8ed1ab_0        conda-forge
    click                          8.1.7           unix_pyh707e725_0   conda-forge
    cloudpickle                    3.0.0           pyhd8ed1ab_0        conda-forge
    defusedxml                     0.7.1           pyhd8ed1ab_0        conda-forge
    doit                           0.36.0          pyhd8ed1ab_0        conda-forge
    empack                         4.0.2           pyhd8ed1ab_0        conda-forge
    entrypoints                    0.4             pyhd8ed1ab_0        conda-forge
    packaging                      24.1            pyhd8ed1ab_0        conda-forge
    pandocfilters                  1.5.0           pyhd8ed1ab_0        conda-forge
    pip                            24.2            pyh8b19718_1        conda-forge
    pkgutil-resolve-name           1.3.10          pyhd8ed1ab_1        conda-forge
    platformdirs                   4.3.6           pyhd8ed1ab_0        conda-forge
    prometheus_client              0.21.0          pyhd8ed1ab_0        conda-forge
    ptyprocess                     0.7.0           pyhd3deb0d_0        conda-forge
    pycparser                      2.22            pyhd8ed1ab_0        conda-forge
    pygments                       2.18.0          pyhd8ed1ab_0        conda-forge
    pysocks                        1.7.1           pyha2e5f31_6        conda-forge
    python                         3.12.6          hc5c86c4_1_cpython  conda-forge
    python-dateutil                2.9.0           pyhd8ed1ab_0        conda-forge
    python-fastjsonschema          2.20.0          pyhd8ed1ab_0        conda-forge
    python-json-logger             2.0.7           pyhd8ed1ab_0        conda-forge
    python_abi                     3.12            5_cp312             conda-forge
    pyyaml                         6.0.2           py312h66e93f0_1     conda-forge
    pyzmq                          26.2.0          py312hbf22597_2     conda-forge
    readline                       8.2             h8228510_1          conda-forge
    referencing                    0.35.1          pyhd8ed1ab_0        conda-forge
    requests                       2.32.3          pyhd8ed1ab_0        conda-forge
    rfc3339-validator              0.1.4           pyhd8ed1ab_0        conda-forge
    rfc3986-validator              0.1.1           pyh9f0ad1d_0        conda-forge
    rich                           13.8.1          pyhd8ed1ab_0        conda-forge
    rpds-py                        0.20.0          py312h12e396e_1     conda-forge
    send2trash                     1.8.3           pyh0d859eb_0        conda-forge
    setuptools                     75.1.0          pyhd8ed1ab_0        conda-forge
    shellingham                    1.5.4           pyhd8ed1ab_0        conda-forge
    six                            1.16.0          pyh6c4a22f_0        conda-forge
    sniffio                        1.3.1           pyhd8ed1ab_0        conda-forge
    soupsieve                      2.5             pyhd8ed1ab_1        conda-forge
    terminado                      0.18.1          pyh0d859eb_0        conda-forge
    tinycss2                       1.3.0           pyhd8ed1ab_0        conda-forge
    tk                             8.6.13          noxft_h[484](https://github.com/talmo/sleap-io-notebook/actions/runs/11089260612/job/30810170200#step:4:489)5f30_101  conda-forge
    tornado                        6.4.1           py312h66e93f0_1     conda-forge
    traitlets                      5.14.3          pyhd8ed1ab_0        conda-forge
    typer                          0.12.5          pyhd8ed1ab_0        conda-forge
    typer-slim                     0.12.5          pyhd8ed1ab_0        conda-forge
    typer-slim-standard            0.12.5          hd8ed1ab_0          conda-forge
    types-python-dateutil          2.9.0.20240906  pyhd8ed1ab_0        conda-forge
    typing-extensions              4.12.2          hd8ed1ab_0          conda-forge
    typing_extensions              4.12.2          pyha770c72_0        conda-forge
    typing_utils                   0.1.0           pyhd8ed1ab_0        conda-forge
    tzdata                         2024a           h8827d51_1          conda-forge
    uri-template                   1.3.0           pyhd8ed1ab_0        conda-forge
    urllib3                        2.2.3           pyhd8ed1ab_0        conda-forge
    webcolors                      24.8.0          pyhd8ed1ab_0        conda-forge
    webencodings                   0.5.1           pyhd8ed1ab_2        conda-forge
    websocket-client               1.8.0           pyhd8ed1ab_0        conda-forge
    wheel                          0.44.0          pyhd8ed1ab_0        conda-forge
    xz                             5.2.6           h166bdaf_0          conda-forge
    yaml                           0.2.5           h7f98852_2          conda-forge
    zeromq                         4.3.5           ha4adb4c_5          conda-forge
    zipp                           3.20.2          pyhd8ed1ab_0        conda-forge
    zstandard                      0.23.0          py312hef9b889_1     conda-forge
    zstd                           1.5.6           ha6fb4c9_0          conda-forge

Trying to pin to conda-forge::sleap-io=0.1.9, which I pushed today to drop av as a hard dependency, I get a different package resolution error:

Looking for: ['python=3.11', 'xeus-python', 'ipycanvas', 'conda-forge::sleap-io=0.1.9', 'emscripten-abi=3.1.45']

https://repo.mamba.pm/emscripten-forge/emscripten-wasm32          Using cache
https://repo.mamba.pm/emscripten-forge/noarch               Using cache
conda-forge/emscripten-wasm32                               Using cache
conda-forge/noarch                                          Using cache
conda-forge/emscripten-wasm32                               Using cache
conda-forge/noarch                                          Using cache
Could not solve for environment specs
The following package could not be installed
└─ sleap-io 0.1.9*  is uninstallable because it requires
   └─ opencv  , which does not exist (perhaps a missing channel).
[LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus] [ERR] Command '['/Users/talmo/mambaforge3/condabin/mamba', 'install', '--yes', '--prefix', PosixPath('/var/folders/qg/bswb6q8x43b0f8gvr4wq21nm0000gn/T/tmpffcb0qls/env/envs/xeus-python-kernel'), '-c', 'https://repo.mamba.pm/emscripten-forge', '-c', 'conda-forge', 'python=3.11', 'xeus-python', 'ipycanvas', 'conda-forge::sleap-io=0.1.9', 'emscripten-abi=3.1.45']' returned non-zero exit status 1.

(Note: this second log is from running locally in my mambaforge environment, which shouldn't matter much here? Unclear...)

Figuring out compatible packages

So far it's clear that at least these packages aren't supported:

• av
• opencv
• ffmpeg

If I use mamba repoquery depends -c conda-forge sleap-io --recursive -p noarch, I can see that there are many deps that should be available, but not all:

 Name                           Version         Build        Channel           
────────────────────────────────────────────────────────────────────────────────
 sleap-io                       0.1.9           pyhd8ed1ab_0 conda-forge/noarch
 numpy >>> NOT FOUND <<<                                                       
 pandas >>> NOT FOUND <<<                                                      
 attrs                          17.2.0          py_1         conda-forge/noarch
 python >>> NOT FOUND <<<                                                      
 imageio                        2.13.1          pyhd8ed1ab_0 conda-forge/noarch
 pillow >>> NOT FOUND <<<                                                      
 opencv >>> NOT FOUND <<<                                                      
 simplejson >>> NOT FOUND <<<                                                  
 ffmpeg >>> NOT FOUND <<<                                                      
 hdmf                           0.post0.dev1081 py_0         conda-forge/noarch
 requests                       2.23.0          pyh8c360ce_2 conda-forge/noarch
 certifi                        2024.8.30       pyhd8ed1ab_0 conda-forge/noarch
 idna                           2.9             py_1         conda-forge/noarch
 chardet                        3.0.4           py_1         conda-forge/noarch
 urllib3                        1.25.9          py_0         conda-forge/noarch
 certifi                        2022.6.15       pyhd8ed1ab_1 conda-forge/noarch
 idna                           3.9             pyhd8ed1ab_0 conda-forge/noarch
 pysocks                        1.7.1           pyha2e5f31_6 conda-forge/noarch
 __unix >>> NOT FOUND <<<                                                      
 pyopenssl                      24.2.1          pyhd8ed1ab_2 conda-forge/noarch
 cryptography >>> NOT FOUND <<<                                                
 brotlipy >>> NOT FOUND <<<                                                    
 scipy >>> NOT FOUND <<<                                                       
 setuptools                     65.3.0          pyhd8ed1ab_1 conda-forge/noarch
 six                            1.14.0          py_1         conda-forge/noarch
 python-dateutil                2.7.0           py_0         conda-forge/noarch
 h5py >>> NOT FOUND <<<                                                        
 ruamel.yaml >>> NOT FOUND <<<                                                 
 pynwb                          0.2.0           py_0         conda-forge/noarch
 imageio-ffmpeg                 0.5.1           pyhd8ed1ab_0 conda-forge/noarch
 ndx-pose                       0.1.1           pyhd8ed1ab_0 conda-forge/noarch
 hdmf                           3.9.0           pyhff52083_0 conda-forge/noarch
 jsonschema                     4.23.0          pyhd8ed1ab_0 conda-forge/noarch
 attrs                          24.2.0          pyh71513ae_0 conda-forge/noarch
 importlib_resources            6.4.5           pyhd8ed1ab_0 conda-forge/noarch
 zipp                           3.20.2          pyhd8ed1ab_0 conda-forge/noarch
 pkgutil-resolve-name           1.3.10          pyhd8ed1ab_1 conda-forge/noarch
 jsonschema-specifications      2023.7.1        pyhd8ed1ab_0 conda-forge/noarch
 referencing                    0.35.1          pyhd8ed1ab_0 conda-forge/noarch
 rpds-py >>> NOT FOUND <<<                                                     
 pynwb                          2.8.2           pyhb401068_0 conda-forge/noarch
 python-dateutil                2.9.0           pyhd8ed1ab_0 conda-forge/noarch
 six                            1.16.0          pyh6c4a22f_0 conda-forge/noarch
 hdmf                           3.14.4          pyhb401068_0 conda-forge/noarch

Filtering to the ones that say >>> NOT FOUND <<<:

 numpy >>> NOT FOUND <<<                                                       
 pandas >>> NOT FOUND <<<                                                      
 python >>> NOT FOUND <<<                                                      
 pillow >>> NOT FOUND <<<                                                      
 opencv >>> NOT FOUND <<<                                                      
 simplejson >>> NOT FOUND <<<                                                  
 ffmpeg >>> NOT FOUND <<<                                                      
 __unix >>> NOT FOUND <<<                                                      
 cryptography >>> NOT FOUND <<<                                                
 brotlipy >>> NOT FOUND <<<                                                    
 scipy >>> NOT FOUND <<<                                                       
 h5py >>> NOT FOUND <<<                                                        
 ruamel.yaml >>> NOT FOUND <<<                                                 
 rpds-py >>> NOT FOUND <<<  

Confusingly, several of these (numpy, pandas, scipy) are installable in different examples mentioned here and elsewhere.

I've noticed some comments about emscripten-forge not being well supported across the ecosystem, but trying to do a package search anyway with mamba repoquery depends -c https://repo.mamba.pm/emscripten-forge -c conda-forge --recursive -p noarch -p emscripten-wasm32 sleap-io does seem to provide something:

Name                                    Version         Build                Channel                                         
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 sleap-io                                0.1.9           pyhd8ed1ab_0         conda-forge/noarch                              
 numpy                                   1.25.2          py311he9a3d86_2      repo.mamba.pm/emscripten-forge/emscripten-wasm32
 python                                  3.11.3          h_bbba66e_25_cpython repo.mamba.pm/emscripten-forge/emscripten-wasm32
 pip                                     20.0.2          py_2                 conda-forge/noarch                              
 setuptools                              65.3.0          pyhd8ed1ab_1         conda-forge/noarch                              
 python                                  3.12.4          h_e6699c1_25_cpython repo.mamba.pm/emscripten-forge/emscripten-wasm32
 emscripten-abi                          3.1.58          h267e887_4           repo.mamba.pm/emscripten-forge/noarch           
 wheel                                   0.34.2          py_1                 conda-forge/noarch                              
 solvable:prereqmarker >>> NOT FOUND <<<                                                                                      
 emscripten-abi                          3.1.45          h7b187fe_15          repo.mamba.pm/emscripten-forge/noarch           
 pandas                                  1.5.3           h5aab7b2_0           repo.mamba.pm/emscripten-forge/emscripten-wasm32
 python-dateutil                         2.7.0           py_0                 conda-forge/noarch                              
 six                                     1.14.0          py_1                 conda-forge/noarch                              
 pytz                                    2017.3          py_1                 conda-forge/noarch                              
 python-tzdata                           2020.1          pyh9f0ad1d_0         conda-forge/noarch                              
 attrs                                   17.2.0          py_1                 conda-forge/noarch                              
 imageio                                 2.13.1          pyhd8ed1ab_0         conda-forge/noarch                              
 pillow                                  10.0.1          h12c5105_0           repo.mamba.pm/emscripten-forge/emscripten-wasm32
 opencv >>> NOT FOUND <<<                                                                                                     
 simplejson >>> NOT FOUND <<<                                                                                                 
 ffmpeg >>> NOT FOUND <<<                                                                                                     
 hdmf                                    0.post0.dev1081 py_0                 conda-forge/noarch                              
 requests                                2.23.0          pyh8c360ce_2         conda-forge/noarch                              
 certifi                                 2024.8.30       pyhd8ed1ab_0         conda-forge/noarch                              
 idna                                    2.9             py_1                 conda-forge/noarch                              
 chardet                                 3.0.4           py_1                 conda-forge/noarch                              
 urllib3                                 1.25.9          py_0                 conda-forge/noarch                              
 certifi                                 2022.6.15       pyhd8ed1ab_1         conda-forge/noarch                              
 idna                                    3.9             pyhd8ed1ab_0         conda-forge/noarch                              
 pysocks                                 1.7.1           pyha2e5f31_6         conda-forge/noarch                              
 __unix >>> NOT FOUND <<<                                                                                                     
 pyopenssl                               24.2.1          pyhd8ed1ab_2         conda-forge/noarch                              
 cryptography                            42.0.2          py311h1562a1a_3      repo.mamba.pm/emscripten-forge/emscripten-wasm32
 cffi                                    1.15.1          0_pypy39             conda-forge/noarch                              
 python_abi >>> NOT FOUND <<<                                                                                                 
 python >>> NOT FOUND <<<                                                                                                     
 pypy3.9 >>> NOT FOUND <<<                                                                                                    
 brotlipy >>> NOT FOUND <<<                                                                                                   
 scipy                                   1.11.1          h6f59b9d_1           repo.mamba.pm/emscripten-forge/emscripten-wasm32
 openblas                                0.3.23          h6f59b9d_2           repo.mamba.pm/emscripten-forge/emscripten-wasm32
 h5py >>> NOT FOUND <<<                                                                                                       
 ruamel.yaml >>> NOT FOUND <<<                                                                                                
 pynwb                                   0.2.0           py_0                 conda-forge/noarch                              
 imageio-ffmpeg                          0.5.1           pyhd8ed1ab_0         conda-forge/noarch                              
 ndx-pose                                0.1.1           pyhd8ed1ab_0         conda-forge/noarch                              
 hdmf                                    3.9.0           pyhff52083_0         conda-forge/noarch                              
 numpy                                   2.0.1           py312h4e66579_0      repo.mamba.pm/emscripten-forge/emscripten-wasm32
 pandas                                  2.2.2           np125py311he4605d8_0 repo.mamba.pm/emscripten-forge/emscripten-wasm32
 jsonschema                              4.23.0          pyhd8ed1ab_0         conda-forge/noarch                              
 attrs                                   24.2.0          pyh71513ae_0         conda-forge/noarch                              
 importlib_resources                     6.4.5           pyhd8ed1ab_0         conda-forge/noarch                              
 zipp                                    3.20.2          pyhd8ed1ab_0         conda-forge/noarch                              
 pkgutil-resolve-name                    1.3.10          pyhd8ed1ab_1         conda-forge/noarch                              
 jsonschema-specifications               2023.7.1        pyhd8ed1ab_0         conda-forge/noarch                              
 referencing                             0.35.1          pyhd8ed1ab_0         conda-forge/noarch                              
 rpds-py >>> NOT FOUND <<<                                                                                                    
 pynwb                                   2.8.2           pyhb401068_0         conda-forge/noarch                              
 python-dateutil                         2.9.0           pyhd8ed1ab_0         conda-forge/noarch                              
 six                                     1.16.0          pyh6c4a22f_0         conda-forge/noarch                              
 hdmf                                    3.14.4          pyhb401068_0         conda-forge/noarch                              

Which just leaves these as missing:

 solvable:prereqmarker >>> NOT FOUND <<<                                                                                      
 opencv >>> NOT FOUND <<<                                                                                                     
 simplejson >>> NOT FOUND <<<                                                                                                 
 ffmpeg >>> NOT FOUND <<<                                                                                                     
 __unix >>> NOT FOUND <<<                                                                                                     
 python_abi >>> NOT FOUND <<<                                                                                                 
 python >>> NOT FOUND <<<                                                                                                     
 pypy3.9 >>> NOT FOUND <<<                                                                                                    
 brotlipy >>> NOT FOUND <<<                                                                                                   
 h5py >>> NOT FOUND <<<                                                                                                       
 ruamel.yaml >>> NOT FOUND <<<                                                                                                
 rpds-py >>> NOT FOUND <<<     

Is this right? I'm surprised h5py wouldn't be available somehow, but I'm not really sure where to go from here...

Questions

1. What is the recommended workflow for making a JupyterLite + xeus-python compatible environment?

• Right now it seems I just have to recurse through all my dependencies and dependencies's dependencies checking if they are available either (a) as noarch on conda-forge, or (b) on emscripten-forge.
• It's not clear what the right tool is for checking whether the environment would resolve (mamba? micromamba? jupyter lite build ...?).

2. Is there a centralized effort for porting packages for JupyterLite + xeus-python compatibility?

• I can't seem to find any info on whether this is being worked on for the different packages mentioned above, or what the blockers would be for those so I can understand the timeline/feasibility of packages complex packages like opencv.
• It's also not clear to me if this is a question best suited for jupyter-xeus/xeus-python, jupyterlite/xeus, jupyterlite/jupyterlite, emscripten-forge/empack, emscripten-forge/recipes or ???

3. The README makes it seem like the only requirement for a new package is being on conda-forge or emscripten-forge, but shouldn't it say something about dependencies?

• Right now, the README tells us to just check beta.mamba.pm to see if the package is noarch.
• Our package doesn't even appear on beta.mamba.pm even though it's on conda-forge.
• Even if it did appear, judging from packages that are there (example), I don't see how I would easily check that all of its deps are also noarch.
• Even if I could check that all conda-forge deps are noarch, it's not like it crosschecks with emscripten-forge which I'm guessing would work if it's been manually ported?
• The investigation workflow above gets close with mamba repoquery depends ... but it's not clear if that's the way that this whole stack would resolve the environment.

Thanks for the work on this project! We're really excited to be able to use it! 😄