ipywidgets.Output has stopped working in voici (still works in jupyterlite though)

[LiamBindle]

Description

I recently upgraded to xeus-python=0.17 because 0.16 stopped working for me and now I'm having trouble getting ipywidgets.Output() to work. When I update an output widget nothing happens. I think the bug is stems from voici because when I build the notebook with jupyterlite ipywidgets.Output() works as expected.

I created a reproducer here

import ipywidgets
from datetime import datetime

output = ipywidgets.Output()
button = ipywidgets.Button(description='Click me')

display(button)
display(output)

def on_click(*args):
    output.clear_output()
    
    with output:
        print(datetime.now().isoformat())
button.on_click(on_click)

With jupyterlite (expected behaviour)

With voici (nothing happens)

Reproduce

You can build the reproducer like so (I also included a build-environment.yml within...I'm using voici 0.7)

git clone https://github.com/LiamBindle/voici-ipywidget-output-bug-reproducer.git
cd voici-ipywidget-output-bug-reproducer
voici build --contents content --apps lab --apps retro

Serve the files

cd _output
python -m http.server 8010

See that ipywidgets.Output() works in jupyterlite: http://0.0.0.0:8010/lab/index.html

See that ipywidgets.Output() doesn't work in voici: http://0.0.0.0:8010/voici/render/content/reproducer.html?

---

Any thoughts? Thanks in advance!

[LiamBindle]

CCing @martinRenou, @DerThorsten, and @jtpio

[LiamBindle]

#121 might be related, but I tried it's reproducer also exhibited the bug in jupyter lite (whereas #127 is isolated to voici).

I've also confirmed that the traitlet is firing by adding a GET request upon the button click (I can see the GET request firing in the network tab), so it seems this issue is related to the redraw.

[DerThorsten]

@LiamBindle unfortunately I dont know much about voici, but hope that @martinRenou or @jtpio can help

[martinRenou]

There shouldn't be anything specific to Voici. The only thing that comes to my mind is that voici was requiring the alpha version of jupyterlite-xeus https://github.com/voila-dashboards/voici/blob/main/python/voici/pyproject.toml#L36 which was having some issues recently.

I just made a final release of jupyterlite-xeus. Can you try again a new build and see if you still see the issue?

[LiamBindle]

Thanks @DerThorsten and @martinRenou

Hmm, strange that there should be anything unique to voici. I'm definitely seeing the bug in voici but not jupyterlite. That seems like a good lead.

I'll try the new jupyterlite-xeus right now and get back to you

[LiamBindle]

@martinRenou I'm afraid upgrading to jupyterlite-xeus=2.0.0 didn't fix the problem. I've double checked that my build environment is indeed using jupyterlite-xeus=2.0.0. Here's a comparison of the jupyter lite vs voici console log.

Console log with Jupyter Lite (works)

Console log with Voici (doesn't work)

The main differences I see are:

1. The voici log has a bunch of warnings about unsatisfied versions

   Example

   Unsatisfied version 4.3.4 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/apputils (required ^4.3.5)

2. The voici log has warning about failing to fetch ipywidgets through the "jupyter.widget.control" comm channel

   Example

   Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm did not respond in time

Do you have any insight? I'm happy to keep digging but I'd benefit if someone could point me in the right direction.

[trungleduc]

@LiamBindle are you using the latest version of ipywidgets? could you try some previous versions? There are some changes in the newest version that might affect voici.

[LiamBindle]

@trungleduc Thanks for the suggestion. I have tried earlier versions of 8 and I get the same problem. In fact, I've tried pinning all packages (including dependencies) to versions that were released prior to June 2024 and that doesn't work either, but I know these versions used to work with voici because we exercise them pretty heavily.

The only package that I haven't been able to pin to a pre-June version is xeus-python because v0.16 seems to have stopped working due to an error creating the default directories when xpython is initializing (I'm not sure why this error wasn't an issue previously).

Any ideas?

[datakurre]

@LiamBindle Have you found the reason? I probably have the same issue, found this and also tried out your reproducer with the same issue still there.

[LiamBindle]

@datakurre I'm afraid not. I spent a day or so trying to figure out what was happening, but I don't think I can dig deeper without a lead. I'm pursuing a workaround for our applications (switching to voila).

If anyone has any insight I'd be keen to dig deeper.

[datakurre]

@LiamBindle Would you have a any working builds where I could get “last known working set” of versions to try? Maybe even for the previous xeus-python?

[LiamBindle]

@datakurre I believe the previous versions have stopped working. My deployments with xeus-python=0.16 were working in July, but since then the versions that I had pinned stopped working in new deployments (errors related to failing to create default directories for Python). This prompted me to switch to xeus-python=0.17 which unveiled this issue.

Here's an environment file that I used May-July 2024, so I know these pinned versions used to worked.

name: voici
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - xeus-python=0.16.0
  - xeus-python-shell=0.6.1
  - pandas<2
  - ipyleaflet=0.19.0
  - ipywidgets=8.1.2
  - ipython=8.23.0  # see https://github.com/emscripten-forge/recipes/issues/927
  - pystac=1.10.0
  - urllib3>=2.2
  - requests<2.31
  - matplotlib=3.8.4
  - plotly<5.22.0
  - numpy=1.25.2
  - openpyxl<3.1.4

That's why it's a bit of an unfortunate situation---the current version has this bug, and previous versions have stopped working.

Edit: Just want to mention that the docs say voici is experimental, so I understand that stuff like this is the risk we take by depending on experimental projects. I'm super grateful for all the Jupyter team's work

[datakurre]

@LiamBindle Thank you for your environment file. I can confirm that I got the same issues with it.

But I was wondering, if you still had a working build from May-July, maybe you could find more comprehensive version list there? For example, file listing from ./xeus/kernel_packages or contents of ./xeus/kernels/xpython/empack_env_meta.json from the voici output directory?

I do share the gratitude and excitement for this project!

[LiamBindle]

@datakurre Ah good idea! Here is empack_env_meta.json from a deployment that looks like it's still working (note that alcesflow is our package).

Let me know if you have success pinning old versions.

[datakurre]

@LiamBindle Thank you. I did try those versions in environment.yml, but unfortunately the results were the same. Both xeus-python = 0.16.0 and 0.17.1 failed similarly. Possibly I should try those versions with older Voici-releases later 🤔

[LiamBindle]

@datakurre Thanks for trying. Your findings are consistent with mine. It seems like the existing binaries changed somehow, or there some browser thing broke the old packages. I did try an older version of voici but got the same result. I'm confused too

[Justyouraveragehomie]

Hi, I'm new to Voici. Just chiming in to mention I am also getting the same/similar errors as above:

Unsatisfied version 0.4.0 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlite/kernel (required ~0.4.4)
Unsatisfied version 0.4.0 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlite/contents (required ~0.4.4)

Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm did not respond in time

[Justyouraveragehomie]

This seems related: jupyter-widgets/ipywidgets#3633

[datakurre]

Is this related to the issue fixed for pyodide-kernel in https://github.com/jupyterlite/pyodide-kernel/pull/148/files ?

I tried to build a project with pyodide-kernel instead, but failed with a completely unrelated build issue

error    libmamba Could not solve for environment specs
    The following package could not be installed
    └─ jupyterlite-pyodide-kernel 0.4.7**  is not installable because it requires
       └─ jupyterlite-core >=0.4.5,<0.5.0 , which requires
          └─ jupyter_core >=4.7  but there are no viable options
             ├─ jupyter_core 5.7.2 would require
             │  └─ __unix, which is missing on the system;
             └─ jupyter_core 5.7.2 would require
                └─ __win, which is missing on the system.

[jtpio]

@datakurre yeah it looks like a different issue when resolving the build environment. Feel free to open a different issue or discussion if needed, thanks!

[datakurre]

Is this related to the issue fixed for pyodide-kernel in https://github.com/jupyterlite/pyodide-kernel/pull/148/files ?

I've been slowly learning to hack jupyterlite-xeus, but would anyone have ideas, what should I look into or try with it?

If I understand correctly, in pyodide-kernel a similar issue was fixed, but I still don't understand if that fix could be applied to jupyterlite-xeus (due to different architecture) or if this is even the same issue.

I assume that jupyter.widget.control related warnings are just symptoms of the original issue.

It is also still a mystery, why we are not able to find old set working versions 🤔

(My libmamba issue was just me.)

[Justyouraveragehomie]

I was thinking that https://github.com/deathbeds/jupyterlite-pyodide-lock/ might help to load ipywidgets (or other packages) in the pyodide kernel without using await/micropip, would work like the xeus kernel where the packages are included. I am not sure if the project is ready to work with Voici yet however.

[datakurre]

I was thinking that https://github.com/deathbeds/jupyterlite-pyodide-lock/ might help to load ipywidgets (or other packages) in the pyodide kernel without using await/micropip, would work like the xeus kernel where the packages are included.

That sounds interesting. Thanks. I assume that it would require some customization into voici build, it could still be easier to resolve than this mystery issue 🤔

[datakurre]

Until we are able to get Voici working with ipywidgets again, is there ways to may JupyterLite look more alike it: only show the notebook, hide code cells, auto run on load?

[jtpio]

Until we are able to get Voici working with ipywidgets again, is there ways to may JupyterLite look more alike it: only show the notebook, hide code cells, auto run on load?

It should technically be possible (maybe even via a JupyterLab extension), but something to track in a separate issue?

[datakurre]

Until we are able to get Voici working with ipywidgets again, is there ways to may JupyterLite look more alike it: only show the notebook, hide code cells, auto run on load?

It should technically be possible (maybe even via a JupyterLab extension), but something to track in a separate issue?

Yes! I was expecting that such extensions would already exist, but if not, let's discuss it at jupyterlite/jupyterlite#1556

[datakurre]

Ok. I learned my lesson to not try reinvent the wheel, but stay with voici.

I got voici working again with pyodide-kernel, thanks to Nick Bollweg, with:

• voici with fixed pyodide-kernel support emulate upstream config-utils relative path rewriting #138
• https://github.com/deathbeds/jupyterlite-pyodide-lock to lock and prefetch ipywidgets and other dependencies

This was still with patched voici 0.7.1 and jupyterlite 0.4.5. I have not tried the latest versions yet. I first created a jupyterlite project with pyodide-lock and once my notebooks were working, I added patched voici and built with voici build.