Raspberry Pi OS Bookworm + Pep 668

[Gadgetoid]

⚠️ Progress / TODO List moved to #16

Rationale

Update:

• I have switched to ~/.virtualenvs/pimoroni' as the default virtualenv dir, since this is what virtualenvwrapper` uses, and thus is one of the search paths used by Visual Studio Code.
• I have raised a PR with Thonny to get this same directory supported there, so we can have some cohesion between CLI, VSCode, Thonny - cpython_frontend: Support for ~/.virtualenvs directory on unix. thonny/thonny#2978

As of RPi OS Bookworm (required for the Raspberry Pi 5) incorporating Pep 668, we are required to install Python packages into a virtual environment or "venv".

The TLDR of 668 is twofold -

• Allowing users to affect the system-wide Python installation could lead to broken system packages or a broken system Python environment. Therefore we should not.
• Using a virtual environment allows dependencies to be managed per project, so the chance of conflicting dependencies (one library needs v1.0.0 another needs v2.0.0 for example) is reduced.

Attempting to modify the system Python environment will kick out an error along the lines of:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
  python3-xyz, where xyz is the package you are trying to
  install.

Raspberry Pi's own documentation (linked in this error message on RPi OS) attempts to cover some basic use cases of virtual environments ans shows some per-project and per user patterns you might consider if you're keen to learn.

For most beginners, however, the specifics of setting up a virtual environment are largely meaningless and due to the non-destructive - you might say "virtual" - nature of these environments there is no harm in us stepping in as the arbiter of truth and setting up one by default.

We can do this in a couple of ways:

Supporting venvs in our Python boilerplate

The changes in this PR attempt to detect and (optionally) manage a virtual environment for the user.

The virtual environment we use by default will live in $HOME/.virtualenvs/pimoroni and this will be shared across all packages, giving a single common mixed use venv for beginners.

Providing a simple "manage me a venv" script

It's possible for a user to opt to gloss over the existence of a virtual environment by adding the following lines to ~/.bashrc -

PY_ENV_DIR=$HOME/.virtualenvs/pimoroni
if [ ! -f $PY_ENV_DIR/bin/activate ]; then
  printf "Creating user Python environment in $PY_ENV_DIR, please wait...\n"
  mkdir -p $PY_ENV_DIR
  python3 -m venv --system-site-packages $PY_ENV_DIR
fi
printf " ↓ ↓ ↓ ↓   Hello, we've activated a Python venv for you. To exit, type \"deactivate\".\n"
source $PY_ENV_DIR/bin/activate

This will create (if it does not exist) and activate a Python virtual environment automatically any time a bash shell is opened. If the user doesn't want to use it, they can type "deactivate" or switch into another virtual environment just as easily.

The idea behind this is that users who have installed our software, and are set up to use our $HOME/.virtualenvs/pimoroni virtual environment don't need to explicitly activate it every time they want to work with our products.

Putting this into ~/.bashrc automatically would be unwise since:

• It's a little rude, since it's difficult to explain in crystal clear detail just how this will modify the users shell
• It's quite difficult to check if it's already there
• It's difficult to automatically update, should it ever need changing
• We don't know what else is in ~/.bashrc and how it might (negatively) interact with this snippet
• There are many instances where it doesn't help or take effect; ie: VSCode or Thonny.

But it's probably useful for users who really don't want to have to think about virtual environments (much.)

System Packages

In both cases, since we still rely on apt packages to satisfy some dependencies, we're using --system-site-packages to pass through the system Python packages into the virtualenv. This is generally unwise since it introduces packages we can't properly manage into the virtualenv, but for now it permits the use of gpiozero + lgpio (the latter missing a proper pypi package) and 'libgpiod' (also currently missing a working pypi package),

What about my precious rpi_ws281x?

In order to run Python code as root within the virtualenv context you need to:

sudo --preserve-env=PATH python my_code.py

In conclusion

Many users don't really want to know or care about a per-project virtual Python environment, and just want a close approximation of the old behaviour. This gets us part of the way with the notable exceptions that:

• Running Python code as root is more complicated (see above)
• System-owned Python packages are made available to the virtual environment, but this may cause conflicts between older (RPi OS ships with a 3 year old libgpiod 1.6.3) system packages a newer Pypi packages.
• It's not always obvious how to use a virtual env with a particular IDE, Thonny's UX around venvs- for example - is, confusing.

[github-actions]

Pull Request Test Coverage Report for Build 7476616755

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 91.667%

---

Totals
Change from base Build 5880039616:	0.0%
Covered Lines:	11
Relevant Lines:	12

---

💛 - Coveralls

[aallan]

Raspberry Pi's own documentation (linked in this error message on RPi OS) attempts to cover some basic use cases of virtual environments ans shows some per-project and per user patterns you might consider if you're keen to learn.

If you have anything that should go into that documentation, which as you mention is linked right in the error message, please either drop me email or raise a PR against the develop branch on the docs repo.

[Gadgetoid]

linked right in the error message

The two people who actually read error messages will appreciate that 😭

Right now I don't have any grand designs on the documentation, my preference is mostly to smooth over the transition as much as possible via our installers (due to aforementioned and unfortunately not sarcastic problem with people simply not reading errors or documentation).

I will add that it did confuse me a little when I clicked through and went straight to "Python on Raspberry Pi" rather than "About Python virtual environments."

(Thanks for finding your way here, by the way, I assume you followed my whimpering cries of anguish)

[rglidden]

It might be nice to add --prompt Pimoroni or something similar to the creation of the venv so that when the environment is activated, the user's prompt changes to (Pimoroni) pi@raspberrypi:~ $ which at least indicates that they're in a Pimoroni environment. The default prompt of (venv) pi@raspberrypi:~ $ is a little confusing as it's not clear which "venv" you have activated.

[Gadgetoid]

Thank you for taking the time to look at this. I'm keenly aware that foisting something like this upon users should be done at least somewhat carefully, and thus any and all feedback is really, really appreciated.

TLDR: Yes, I totally agree!

[Gadgetoid]

Considering recommending an extra line in ~/.bashrc to make the terminal startup output something like:

 ↓ ↓ ↓ ↓   Hello, we've activated a Python venv for you. To exit, type "deactivate".
(Pimoroni) phil@ubuntu-pi5:~$

[aallan]

linked right in the error message

The two people who actually read error messages will appreciate that 😭

I mean, the folks who read it seem happy afterward, e.g. https://twitter.com/drfootleg/status/1714906054302060692.

Right now I don't have any grand designs on the documentation, my preference is mostly to smooth over the transition as much as possible via our installers (due to aforementioned and unfortunately not sarcastic problem with people simply not reading errors or documentation).

…and I don't disagree.

I will add that it did confuse me a little when I clicked through and went straight to "Python on Raspberry Pi" rather than "About Python virtual environments."

Future-proofing myself, I've got a sneaking suspicion that I'm going to need more verbiage eventually.

(Thanks for finding your way here, by the way, I assume you followed my whimpering cries of anguish)

Pretty much! 😆

[aallan]

Considering recommending an extra line in ~/.bashrc to make the terminal startup output something like:

 ↓ ↓ ↓ ↓   Hello, we've activated a Python venv for you. To exit, type "deactivate".
(Pimoroni) phil@ubuntu-pi5:~$

This sounds like a good plan!

[rbricheno]

This is a noble attempt at helping, which really should be addressed at the OS level. I raised an issue here raspberrypi/bookworm-feedback#100

[Footleg]

My attempts at following the suggestions here, on Bookworm 64bit OS on the Pi 5. First I tried to run the example without sudo (predicably it cannot access the LEDs). Then I try with sudo but still it fails.

(robots-venv) pi@raspberrypi5:~/inventorhatmini-python/examples $ python led_rainbow.py
Traceback (most recent call last):
  File "/home/pi/inventorhatmini-python/examples/led_rainbow.py", line 18, in <module>
    board = InventorHATMini()
            ^^^^^^^^^^^^^^^^^
  File "/home/pi/robots-venv/lib/python3.11/site-packages/inventorhatmini/__init__.py", line 99, in __init__
    GPIO.setup(self.PI_USER_SW_PIN, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
RuntimeError: No access to /dev/mem.  Try running as root!
Exception ignored in: <function InventorHATMini.__del__ at 0x7fff52ecd3a0>
Traceback (most recent call last):
  File "/home/pi/robots-venv/lib/python3.11/site-packages/inventorhatmini/__init__.py", line 142, in __del__
    self.ioe.reset()
    ^^^^^^^^
AttributeError: 'InventorHATMini' object has no attribute 'ioe'
(robots-venv) pi@raspberrypi5:~/inventorhatmini-python/examples $ sudo --preserve-env=PATH python led_rainbow.py
Traceback (most recent call last):
  File "/home/pi/inventorhatmini-python/examples/led_rainbow.py", line 18, in <module>
    board = InventorHATMini()
            ^^^^^^^^^^^^^^^^^
  File "/home/pi/robots-venv/lib/python3.11/site-packages/inventorhatmini/__init__.py", line 99, in __init__
    GPIO.setup(self.PI_USER_SW_PIN, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
RuntimeError: Mmap of GPIO registers failed
Exception ignored in: <function InventorHATMini.__del__ at 0x7fff9a65d3a0>
Traceback (most recent call last):
  File "/home/pi/robots-venv/lib/python3.11/site-packages/inventorhatmini/__init__.py", line 142, in __del__
    self.ioe.reset()
    ^^^^^^^^
AttributeError: 'InventorHATMini' object has no attribute 'ioe'

[Gadgetoid]

Yup RPi.GPIO - which any of our drivers using GPIO will use - is pretty much dead in the water at the moment. That's why I'm embroiled in getting the libgpiod Python bindings ship shape, so I can start migrating drivers to those.

There is an lgpio (which uses libgpiod) based compatibility shim for RPi.GPIO which might help: https://github.com/waveform80/rpi-lgpio

I've been too mired in getting the absolute basics working to see if there's any short term crutch we can rely on, however.

[Gadgetoid]

My latest Pi OS install has a non-link file for /boot/config.txt with the contents:

DO NOT EDIT THIS FILE

The file you are looking for has moved to /boot/firmware/config.txt

Need to check if this is now standard across all Pi OS installs and, in any case, check for this text and avoid emitting this warning.

A simpler path might be to avoid all this kerfuffle and always prefer /boot/firmware/config.txt if it exists, without any efforts to warn about other config files.

Or maybe even fail outright if /boot/firmware/config.txt does not exist...

[Gadgetoid]

Okay I found the change: RPi-Distro/pi-gen@a125344

It will be standard, but a quick test of 64bit Pi OS lite suggests it currently is not. Therefore the symlink warning is obsolete.

[bsimmo]

Just tried growhat on RiOS bookworn 32bit, and it blew up at me, this was one of the problems. It doesn't have anything in /boot/firmware/ at all.

[Gadgetoid]

Grow should have a fallback to /boot/config.txt- do you have a full output of the install script?

[bsimmo]

I'll try again, but nothing was added to the normal config.txt.

For now to get it working I've moved to legacy 'bullseye' and the install was ok.

I'll have another go with bookworm when I get the time. (Then for fun I'll try the EnviroHat and the WeatherHat and the led matrix pi zero sized thing if forget the name of... obviously not as much fun as you're having rewriting all of this for all the devices.

But the venv never seemed to go right either, it didn't boot to one. and then when I manually enabled it none of the modules seemed to be there when running the examples. I'll check what it actually contains properly, it might be a few days though.

(Pi Zero, so has to be the 32bit lite version).

[Gadgetoid]

Currently I'm not enabling the virtual environment by default, though it should install a script into ~/Pimoroni/auto_venv.sh which can be optionally added to ~/.bashrc.

source ~/.virtualenvs/pimoroni/bin/activate should activate the venv manually.