Switch to using Conda lock files to specify requirements and update dependencies

[matt-graham]

Switches from using pip-compile and requirements.txt files to specify Python dependencies to be installed using pip to using conda-lock to generate lock files specifying (platform specific) Conda environments with both Python and non-Python dependencies.

Also currently set to update dependencies to Python 3.11 and Pandas 2.0.0 - we could potentially pin to the current Python (v3.8) / Pandas (v1.2.2) versions instead to split the change over how we're handling dependencies and the actual updating of dependencies.

[matt-graham]

Currently CI not running correctly, I think probably because of a combination of conda not being installed on test runners and the tox version installed being too old (ideally 3.15 plus is needed I think). @tamuri would you be able to install conda and update tox on the test runners at some point?

EDIT: I think I was misreading the log previously - looks like the problem is only with conda not being installed, the installed tox version (3.24.5) seems to be fine.

[tamuri]

Conda has been installed on the runners.

[tamuri]

Also need to install tox-conda...doing that now

[matt-graham]

Interesting, when I ran locally I didn't need to install tox-conda separately because it is listed in the tox.requires section of the tox.ini file and so should be auto provisioned if a compatible version is not already installed. From the most recent run logs it looks like there is still a problem in finding the conda executable which might mean it is not getting added to PATH variable.

[tamuri]

Moving to conda requires sorting the tox config, GHA self-hosted runners, the container used by Azure Batch. It might not be straightforward. So, I have a suggestion: let's hold off installing the packages from conda i.e. simply update the files in requirements to update the dependencies. We can then start benefitting from all the work you've put in to this quickly. Then look at all these moving parts together and rationalise our approach. @giordano will be also be able to contribute to this.

[matt-graham]

I don't think we want to go forward with this one any more so closing.