Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Python 3.11 #14257

Closed
wants to merge 24 commits into from
Closed
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
694d190
Upgrade to Python 3.11
jessicamack Jul 18, 2023
74ed8e9
install Python 3.11 and use explicit version for Python packages
jessicamack Jul 18, 2023
a4a7a38
fix package names
jessicamack Jul 18, 2023
3e3fae9
use specific version of pip
jessicamack Jul 18, 2023
f972c9e
unpin pytest packages versions
jessicamack Jul 18, 2023
1062f6c
remove unnecessary package. update requirements
jessicamack Jul 18, 2023
d04be80
add missing licenses
jessicamack Jul 20, 2023
86440b5
remove unused licenses
jessicamack Jul 20, 2023
056bdb8
change package name and update licenses
jessicamack Jul 20, 2023
afa7fac
reformat test
jessicamack Jul 20, 2023
bd4e0c9
reformat tests
jessicamack Jul 24, 2023
4357ecd
Merge branch 'devel' into upgrade-to-py3.11
jessicamack Aug 1, 2023
6e709b3
fix redis url
jessicamack Aug 2, 2023
3b8ec51
update requirements
jessicamack Aug 2, 2023
c76ac2b
remove requirement
jessicamack Aug 2, 2023
154a0c8
change package name back
jessicamack Aug 2, 2023
985b61b
temporarily skip test
jessicamack Aug 3, 2023
906a1e7
Update test_options.py
jessicamack Aug 9, 2023
9e30556
remove unused import
jessicamack Aug 9, 2023
53a7565
Merge remote-tracking branch 'upstream/devel' into upgrade-to-py3.11
jessicamack Aug 24, 2023
5eb31d4
Update cache url
jessicamack Aug 24, 2023
3d7c357
remove merge conflict text
jessicamack Aug 24, 2023
1d95093
Merge branch 'upgrade-to-py3.11' of https://github.com/jessicamack/aw…
jessicamack Aug 24, 2023
b0e4cc4
fix requirements merge conflict
jessicamack Aug 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
-include awx/ui_next/Makefile

PYTHON := $(notdir $(shell for i in python3.9 python3; do command -v $$i; done|sed 1q))
PYTHON := $(notdir $(shell for i in python3.11 python3; do command -v $$i; done|sed 1q))
SHELL := bash
DOCKER_COMPOSE ?= docker-compose
OFFICIAL ?= no
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@
- name: Install ansible-runner
pip:
name: ansible-runner
executable: pip3.9
executable: pip3
{% endverbatim %}
2 changes: 1 addition & 1 deletion awx/main/credential_plugins/dsv.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

from django.conf import settings
from django.utils.translation import gettext_lazy as _
from thycotic.secrets.vault import SecretsVault
from delinea.secrets.vault import SecretsVault
Copy link
Member

@AlanCoding AlanCoding Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also see #14207, it looks like we have 2 credential plugins that import this library. And multiple open PRs doing some fraction of the upgrade.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should rebase and merge the maintainers PR and then I can rebase and include that work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's good with me, I asked the question in that PR, because we should strive to keep all plugins functional.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had additional conversation in that PR #14207 and left it with a minor request. I'm completely happy to get that in on its own. Knowing that, it would seem appropriate to remove that change from this python upgrade PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind removing it though I think it might cause errors because the lower versions of python-tss-sdk and python-dsv-sdk don't work with 3.11. Do we want to just ignore the errors for the moment since the other PR will be in shortly and address them?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Errors installing them (which breaks all builds), errors when using the plugin, or more subtle hypothetical python version compatibility issues which might not affect us anyway?

The PR #14207 has some clear steps forward, but they have not been responded too. Additionally, upgrading python-dsv-sdk is not attempted by that. Manually testing imports, I have found that upgrading python-dsv-sdk looks fairly straightforward and easy. In both cases, upgrading the library changes the import path.

If those changes really are needed, you could consider taking over that patch from them, including it in here, or doing a quick separate PR where you upgrade those 2 libraries. We have somewhat of a problem that we have a fragile testing situation, for which I have some potential work up at AlanCoding#81. If you want to verify functionality of these plugins after upgrade, you could try the manual instructions at tools/docker-compose/README.md which have been tested quite recently.



dsv_inputs = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,15 +76,15 @@ def test_callback_absent(self, get_related_mock_and_run, job_template):
class TestJobTemplateSerializerGetSummaryFields:
def test_survey_spec_exists(self, test_get_summary_fields, mocker, job_template):
job_template.survey_spec = {'name': 'blah', 'description': 'blah blah'}
with mocker.patch.object(JobTemplateSerializer, '_recent_jobs') as mock_rj:
mock_rj.return_value = []
test_get_summary_fields(JobTemplateSerializer, job_template, 'survey')
mock_rj = mocker.patch.object(JobTemplateSerializer, '_recent_jobs')
mock_rj.return_value = []
test_get_summary_fields(JobTemplateSerializer, job_template, 'survey')

def test_survey_spec_absent(self, get_summary_fields_mock_and_run, mocker, job_template):
job_template.survey_spec = None
with mocker.patch.object(JobTemplateSerializer, '_recent_jobs') as mock_rj:
mock_rj.return_value = []
summary = get_summary_fields_mock_and_run(JobTemplateSerializer, job_template)
mock_rj = mocker.patch.object(JobTemplateSerializer, '_recent_jobs')
mock_rj.return_value = []
summary = get_summary_fields_mock_and_run(JobTemplateSerializer, job_template)
assert 'survey' not in summary

def test_copy_edit_standard(self, mocker, job_template_factory):
Expand Down
12 changes: 6 additions & 6 deletions awx/main/tests/unit/api/test_views.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,13 +110,13 @@ def exclude(self, **kwargs):

with mocker.patch.object(InventoryInventorySourcesUpdate, 'get_object', return_value=obj):
with mocker.patch.object(InventoryInventorySourcesUpdate, 'get_serializer_context', return_value=None):
with mocker.patch('awx.api.serializers.InventoryUpdateDetailSerializer') as serializer_class:
serializer = serializer_class.return_value
serializer.to_representation.return_value = {}
serializer_class = mocker.patch('awx.api.serializers.InventoryUpdateDetailSerializer')
serializer = serializer_class.return_value
serializer.to_representation.return_value = {}

view = InventoryInventorySourcesUpdate()
response = view.post(mock_request)
assert response.data == expected
view = InventoryInventorySourcesUpdate()
response = view.post(mock_request)
assert response.data == expected
Comment on lines +113 to +119
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jessicamack why was this changed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests were throwing an error and this warning about using mocks as a context manager https://pytest-mock.readthedocs.io/en/latest/remarks.html#usage-as-context-manager so I reformatted the tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A long shot, but could this have been related to the test that we eventually disabled? Can you try reverting these changes and then run tests a) with that naughty test disabled and b) with the naughty test enabled? If we can get past tests with this change reverted, and maybe even get past the other test with this change reverted, then we should revert it.



class TestSurveySpecValidation:
Expand Down
2 changes: 1 addition & 1 deletion awxkit/tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ skip_missing_interpreters = true
# skipsdist = true

[testenv]
basepython = python3.9
basepython = python3.11
setenv =
PYTHONPATH = {toxinidir}:{env:PYTHONPATH:}:.
deps =
Expand Down
6 changes: 2 additions & 4 deletions licenses/jaraco-classes.txt → licenses/aiohttp-retry.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might not be synced, but we need to make sure we regen/clean-up the licenses that were changed that we went back on, right? Like we regenerated without the full updates so we should have way fewer license changes now iiuc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like maybe git checkout devel -- $allthoselicensefiles and then regen and then git add $newlicensefiles && git commit, etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump. @jessicamack Do you have a local copy where you fixed all the license files, etc? That seems to be one of the last pieces to at least mark this as complete. That and @relrod feedback. If you want, I can try to do the cleanup.

Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
# As listed on https://pypi.python.org/pypi/irc

The MIT License (MIT)
Copyright (c) <year> <copyright holders>
Copyright (c) 2020 aiohttp_retry Authors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
21 changes: 21 additions & 0 deletions licenses/annotated-types.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The MIT License (MIT)

Copyright (c) 2022 the contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
19 changes: 19 additions & 0 deletions licenses/jsonschema-specifications.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Copyright (c) 2022 Julian Berman

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Binary file removed licenses/jwcrypto-1.4.2.tar.gz
Binary file not shown.
Binary file added licenses/jwcrypto-1.5.0.tar.gz
Binary file not shown.
21 changes: 21 additions & 0 deletions licenses/pydantic-core.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The MIT License (MIT)

Copyright (c) 2022 Samuel Colvin

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
22 changes: 0 additions & 22 deletions licenses/pyrsistent.txt

This file was deleted.

Loading
Loading