Maintaining a development environment

[purva-thakre]

How does everyone here use Jupyter notebook in their development environment?

The discussion in today's Community Call compelled me to create a new virtual environment for the newer dependencies. However, there is an incompatibility when I try to install Jupyter notebook.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
qcs-api-client 0.21.6 requires attrs<22.0.0,>=21.3.0, but you have attrs 23.2.0 which is incompatible.
qcs-api-client 0.21.6 requires httpx<0.24.0,>=0.23.0, but you have httpx 0.27.0 which is incompatible.

I have had this happen in the past as well. #2145 (comment)

Same as last time, I had to use the following to be able to use Jupyter notebook.

pip uninstall traitlets
pip install traitlets==5.13.0
pip install notebook

I am curious to know if there's a better workaround to the issue I tend to run into besides listing Jupyter notebook as a dev dependency.

Edited to add: This workaround is not ideal as it causes other additional issues https://discord.com/channels/764231928676089909/1220012039494308011

[natestemen]

I'm seeing the same error when setting up a fresh environment (i.e. running conda create -n purva python=3.11, conda activate purva, conda install pip, make install and finally pip install notebook). After this, notebook is installed, but fails to properly load the server it starts on my machine when running jupyter notebook. I'm also seeing an error when trying to install jupyterlab.

One potential solution is to install a global version of jupyter notebook and use that instead of a one inside your mitiq virtual environment. If you try this, let me know if it works!

Personally, I use the Jupyter extension for vscode and let it handle the installs. I'm not totally sure where it puts the notebook download, but I haven't run into any problems like this before. I think this extension only installs ipykernel.

[purva-thakre]

One potential solution is to install a global version of jupyter notebook and use that instead of a one inside your mitiq virtual environment. If you try this, let me know if it works!

This works partially. The import paths have to be consistent for the virtual environment and jupyter notebook installation.

In my virtual environment, I can import mitiq when I install jupyter notebook globally. There are some problems with importing other dependencies like cirq and numpy. Will have to mess around with putting all the dependencies on path.

I think the VS Code extension is a better option.

[gachchami]

I have used the current 0.34.0 with dev-requirements.txt without some test packages and notebook 6.4.13
https://github.com/gachchami/qmitra/blob/2c1f8addb860eb4441581cd952afab7cd4e6dea3/requirements.txt

[gachchami]

I do not need an editable version of mitiq for now. The requirements file I created was derived from dev_requirements.txt of the released tag. this was to ensure the project team does not install anything that mitiq will not like in the future.

I MAY have to have an editable version in the future if we encounter any issues/add docs for the UF project we are working on

[natestemen]

If you do not need an editable install, there is potentially an easier way forward for you then others in this thread.

1. Start a fresh virtual environment
2. Run something like pip install mitiq==0.43.0 (with your desired version of mitiq, or no version)
3. Install your other required dependencies (maybe qiskit, pennylane, and/or numpy, etc)
4. run pip freeze > requirements.txt

This will put all the versions of dependencies you installed into the requirements.txt file. Hence you will not need to choose what dependencies come from each project. Pip will handle all that for you!

[gachchami]

Good suggestion.

I did try that route. But I got conflicts with packages that conda installs on default python. specifically with

requests-ntlm 1.1.0

My team also uses qiskit. installing it was causing a whole set of other issues (due to updating to newer versions of qiskit). so I used the dev_requirements.txt to ensure I install all the dependencies that were compatible with mitiq.

For any other packages I would do pip install <package> --dry-run and pipdeptree to see what are the dependencies and could there be any conflicts.

[natestemen]

I got conflicts with packages that conda installs on default python

Would you mind walking me through what causes the problems? I use conda to manage environments and haven't encountered this issue before, but we likely have slightly different workflows here. I want to ensure it's smooth for all download processes :)

qiskit

We don't yet support qiskit 1.0, but hope to in the next release or two!

pipdeptree

cool, TIL!

[gachchami]

I tried to reproduce this. But I think I did not start with a clean env. I probably installed qiskit or jupyter before trying to install mitiq. this could have caused the conflicts when I installed mitiq.

[cosenal]

Here is a report from my recent trip to dependency hell 🔥 Note that I use Jupyterlab instead of Jupyter Notebook, but the conflicts may be similar.

When installing Jupyterlab with pip install jupyterlab in the virtual env, the latest version is installed, that is, jupyterlab-4.1.5. That carries as dependencies httpx and attrs. I used the pipdeptree tool to understand conflicting dependencies (same tool recommended by @gachchami above.)

* cirq-ft==1.3.0
 - cirq-core [required: ==1.3.0, installed: 1.1.0]
* qcs-api-client==0.21.6
 - attrs [required: >=21.3.0,<22.0.0, installed: 23.2.0]
 - httpx [required: >=0.23.0,<0.24.0, installed: 0.27.0]

So far, nothing new from the warning message in @purva-thakre's original post.

The result of pipdeptree -r -p httpx is more interesting.

httpx==0.27.0
├── jupyterlab==4.1.5 [requires: httpx>=0.25.0]
└── qcs-api-client==0.21.6 [requires: httpx>=0.23.0,<0.24.0] 

By looking at the commit history of jupyterlab, It turns out that a dependency on httpx>=0.25.0 was introduced recently, so one could pin jupyterlab==4.0.12, which is the last version without a strict dependency on httpx. However, that doesn't resolve the conflict with attrs, which is more complicated. For that, I tried to use the automatic dependency resolver backed in pip, but it timed out 😬

pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

In any case, it sounds wrong to retro-fit the version of jupyterlab to please the rest of the dependencies. The right way is to move forward with qcs-api-client. Good news is that recent versions of qcs-api-client don't bound attrs version, see this commit.

Here is how mitiq depends on an older version of qcs-api-client.

qcs-api-client==0.21.6
└── pyquil==3.5.4 [requires: qcs-api-client>=0.21.0,<0.22.0]
    └── cirq-rigetti==1.1.0 [requires: pyquil>=3.2.0]
        └── cirq==1.1.0 [requires: cirq-rigetti==1.1.0]
            └── mitiq==0.35.0.dev0 [requires: cirq>=1.0.0,<1.4.0]

In other words, pyquil is the bottleneck at the moment. I will try to upgrade pyquil, and I will make separate post to report back on the progress.

[cosenal]

Tracking pyQuil major dependency bump in #2254

[cosenal]

And in fact it's cirq, and in turns, cirq-rigetti, which is bounding those dependencies.
@natestemen @purva-thakre See discussion in quantumlib/Cirq#6500

[purva-thakre]

@cosenal Nice job with getting out of the dependency hell! I have always ended up giving up halfway out of frustration. :)

Correct me if I am wrong but from what I understand, one possible solution could be:

• separating out the cirq related subpackages in requirements.txt.
• Skip installing cirq-rigetti for now
• Temporarily disable all unit tests related to cirq-rigetti

Cirq becoming compatible with pyQuil v4 won't help us that much until #2254 is closed. It's great that they believe upgrading to v4 will be a quick fix.

From what I see, this dependency hell also occurs if we install a non-editable version of mitiq, right? Then, perhaps we also need to provide some additional information in the docs that a user should still be able to use the jupyter notebook extension in VS Code.

[cosenal]

@purva-thakre Yes, that's one possible solution. However, since Cirq is used for the internal circuit representation, removing cirq-rigetti from the dependencies would mean that Mitiq wouldn't support pyQuil at all.
Good news from my initial investigation on pyQuil 4 is that once Cirq becomes compatible with it, #2254 becomes an easy job for us.

[purva-thakre]

Going to go ahead and close this discussion since we know which dependency is causing this and there's an open issue related to it.