Upgrade Python to 3.10 in app VM and psycopg2 to 2.9 #1378

rachelekm · 2024-03-01T20:18:35Z

Overview

Upgrades the app VM from Python 3.6.9 to 3.10.12 as well as upgrades psycopg2 from 2.8.6 to 2.9 in both app and database VM. Both of these upgrades support upcoming Django and PostgreSQL upgrades in the #1372 task list.

Ideally we would get to Python version 3.12 since that is currently in bugfix maintenance status into 2025, but I ran into a variety of issues trying to upgrade to this version so am leaving it at 3.10.12. This version is still receiving security support through October 2026. (see notes for more detail)

Demo

RUNNING HANDLER [cac-tripplanner.app : Restart nginx] **************************
changed: [app]

PLAY RECAP *********************************************************************
app                        : ok=37   changed=32   unreachable=0    failed=0    skipped=17   rescued=0    ignored=0   

rachelemorino@Racheles-MacBook-Pro cac-tripplanner % vagrant ssh app
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-92-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

  System information as of Fri Mar  1 07:04:49 PM UTC 2024

  System load:  0.39599609375      Processes:             159
  Usage of /:   19.1% of 30.34GB   Users logged in:       0
  Memory usage: 54%                IPv4 address for eth0: 10.0.2.15
  Swap usage:   1%                 IPv4 address for eth1: 192.168.56.24


This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento
Last login: Fri Mar  1 19:03:31 2024 from 192.168.56.1
vagrant@app:~$ python3 --version
Python 3.10.12

vagrant@app:~$ pip show psycopg2-binary
Name: psycopg2-binary
Version: 2.9
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: https://psycopg.org/
Author: Federico Di Gregorio
Author-email: [email protected]
License: LGPL with exceptions
Location: /usr/local/lib/python3.8/dist-packages
Requires:

vagrant@database:~$ pip show psycopg2
Name: psycopg2
Version: 2.9.9
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: https://psycopg.org/
Author: Federico Di Gregorio
Author-email: [email protected]
License: LGPL with exceptions
Location: /usr/local/lib/python3.8/dist-packages
Requires: 
Required-by:

Notes

Context on the nginx role changes:

Upgrading to Python 3.10 failed app provisioning with the following error:

TASK [azavea.nginx : Configure the Nginx PPA] **********************************
fatal: [app]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: E:The repository 'http://ppa.launchpad.net/nginx/stable/ubuntu jammy Release' does not have a Release file."}

After digging it appears upgrading the upgraded Ubuntu version was too modern for the repo defined in the
azavea.nginx role. Unfortunately, this is a separate repo so I had to copy over the entire role and update app's role
dependencies in order to update this task to use the correct stable PPA.

Notes on issues upgrading to Python 3.12:

The Ubuntu 22.04 base box brings the Python default to 3.10.12, but ideally we would upgrade to Python 3.12 to give us the most runway before contract end. I tried to manually install Python 3.12 from PPA as part of app provisioning and then update the ansible_python_interpreter to locate the correct version to use, but ran into incompatibilities with the ansible version:

TASK [azavea.packer : Download Packer] *****************************************
fatal: [app]: FAILED! => {"changed": false, "dest": "/usr/local/src/packer_1.5.4_linux_amd64.zip", "elapsed": 0, "msg": "An unknown error occurred: 'CustomHTTPSConnection' object has no attribute 'cert_file'", "url": "https://releases.hashicorp.com/packer/1.5.4/packer_1.5.4_linux_amd64.zip"}

The upgraded Python version got rid of some variables, like cert_file, that are used by ansible in the azavea.packer role and we would need to upgrade local ansible-core to version >=2.15 for these to be compatible, see this ansible issue for more detail. This project depends on local installation of ansible, but we have historically had issues using upgraded ansible versions beyond 6.0 with this project. The latest version of ansible v 5.0 only includes up to ansible-core v 2.12.7. I could target an update for ansible-core to get these fixes, but I suspect this could open a can of worms and given our remaining budget and that we're able to at least get to Python 3.10 I think we should leave this for down the road. Curious if you think it's worth it!

Testing Instructions

vagrant destroy (Ensure you have Graph.obj in the otp_data dir before destroying the otp VM so it doesn't rebuild the graph on up)
vagrant up (you may need to run CAC_APP_SHARED_FOLDER_TYPE=virtualbox vagrant up)
Confirm all VMs are successfully brought up and app is working as expected in the browser
ssh into the app VM and confirm python3 version and psycopg2-binary version
ssh into the database VM and confirm psycopg2 version

Checklist

No gulp lint warnings
No python lint warnings
Python tests pass
All TODOs have an accompanying issue link

Connects #1356

rachelekm · 2024-03-01T20:20:20Z

Vagrantfile

@@ -86,6 +86,9 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  end

  config.vm.define "database" do |database|
+    # SSH issues bringing up DB VM with base box 22.04
+    # Explicitly define downgraded version for use by database until fix 
+    database.vm.box = "bento/ubuntu-20.04"


When configuring all machines with a "bento/ubuntu-22.04" base box I ran into issues successfully provisioning and bringing up the database VM. As part of booting the VM it would hang on database: SSH auth method: private key before failing with a timeout error. This has been a reported issue on Vagrant before, but all the previous solutions recommended in comments don't work. I previously ran into this issue once with Cicero and I was able to solve it by explicitly defining the ssh auth.username and ssh.auth.password in the Vagrantfile, but that did not work here either.

I settled on configuring a less-upgraded base box that doesn't have ssh issues (20.04) for the database and then configuring the more-upgraded (22.04) base box version for otp and app. I'm not sure if we're going to run into trouble with these two being different, but I struggled to manually install Python 3.10 for use in the Ubuntu 20.04 version so this was the only way I could figure out how to bring Python up past 3.8 in the app while also avoiding VirtualBox issues.

rachelekm · 2024-03-01T22:00:30Z

deployment/ansible/roles/cac-tripplanner.nginx/tasks/main.yml

+# Tasks and related artifacts copied from azavea.nginx role
+# Customized to use Nginx Stable PPA that works with Ubuntu 22.04
+- name: Configure the Nginx PPA
+  apt_repository: repo=ppa:ondrej/nginx state=present


Hard to tell since it's a new file, but line 5 has been changed from apt_repository: repo=ppa:nginx/{{ nginx_version }} state=present. See notes for details on why this was needed.

rachelekm · 2024-03-01T22:02:30Z

deployment/ansible/roles/cac-tripplanner.nginx/defaults/main.yml

@@ -0,0 +1,2 @@
+---
+nginx_delete_default_site: False


In addition to updating the nginx repo in the "Configure the Nginx PPA" task, I removed the line nginx_version: "stable" here in the copied over defaults.yml since it's no longer used by this role.

rajadain

Great work! Lots of detail and detours.

I tried provisioning database with ubuntu-22.04 and ran into the same issue. I suspect it's a VirtualBox Guest Additions mismatch, but I'm not sure why it works for app and otp but not database. Nevertheless, I think it's okay to keep database at an older version for now. It's only used in development, and the database API is pretty consistent.

rachelekm · 2024-03-04T19:40:55Z

Thanks for the review! Added one more commit to clarify how to run migrations following our slack discussion.

rachelekm added 4 commits March 1, 2024 14:46

Upgrade to Pythonv3.8.10 and psycopg2

a3377e0

Update base box to 22.04 for upgraded Python default

d909581

Copy azavea.nginx role over to customize

122967d

Update nginx to Ubuntu 22.04-compatible repo

912102c

rachelekm commented Mar 1, 2024

View reviewed changes

rachelekm marked this pull request as ready for review March 1, 2024 21:56

rachelekm commented Mar 1, 2024

View reviewed changes

rachelekm requested a review from rajadain March 1, 2024 22:08

rachelekm assigned rajadain Mar 1, 2024

rajadain approved these changes Mar 4, 2024

View reviewed changes

rajadain assigned rachelekm and unassigned rajadain Mar 4, 2024

Add migrations note to README for spin-up clarity

778fe61

rachelekm merged commit d1c3c70 into develop Mar 4, 2024

rachelekm deleted the rm/upgrade-python-app-vm branch March 4, 2024 19:41

rachelekm mentioned this pull request Mar 28, 2024

Upgrade to Django 4.2 #1380

Merged

4 tasks

rachelekm mentioned this pull request May 6, 2024

Update AWS Ubuntu version in prod #1384

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Python to 3.10 in app VM and psycopg2 to 2.9 #1378

Upgrade Python to 3.10 in app VM and psycopg2 to 2.9 #1378

rachelekm commented Mar 1, 2024 •

edited

Loading

rachelekm Mar 1, 2024

rachelekm Mar 1, 2024 •

edited

Loading

rachelekm Mar 1, 2024

rajadain left a comment

rachelekm commented Mar 4, 2024

Upgrade Python to 3.10 in app VM and psycopg2 to 2.9 #1378

Upgrade Python to 3.10 in app VM and psycopg2 to 2.9 #1378

Conversation

rachelekm commented Mar 1, 2024 • edited Loading

Overview

Demo

Notes

Context on the nginx role changes:

Notes on issues upgrading to Python 3.12:

Testing Instructions

Checklist

rachelekm Mar 1, 2024

Choose a reason for hiding this comment

rachelekm Mar 1, 2024 • edited Loading

Choose a reason for hiding this comment

rachelekm Mar 1, 2024

Choose a reason for hiding this comment

rajadain left a comment

Choose a reason for hiding this comment

rachelekm commented Mar 4, 2024

rachelekm commented Mar 1, 2024 •

edited

Loading

rachelekm Mar 1, 2024 •

edited

Loading