Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add howto guides for switching from the charm plugin #1967

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions docs/howto/charm-to-poetry.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
.. _howto-migrate-to-poetry:

Migrate from the ``charm`` plugin to Poetry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with existing docs and docs best practices, we should use the form Charm plugin and Python plugin.

===========================================

Many charms use `Poetry`_ to manage their Python projects. For these charms, Charmcraft
has a :ref:`craft_parts_poetry_plugin`. Migrating from the charm plugin provides some
benefits, such as no longer having to maintain a ``requirements.txt`` file. If the
charm to be migrated does not currently use poetry, refer to the
`Poetry documentation <https://python-poetry.org/docs/basic-usage/>`_ for instructions
on how to use poetry for a Python project.

Update ``charmcraft.yaml``
--------------------------

The first step is to update ``charmcraft.yaml`` to include the correct parts definition.
Depending on the history of a specific charm, it may not have an explicitly-included
``parts`` section determining how to build the charm. In this case, a ``parts`` section
can be created as follows:

.. code-block:: yaml

parts:
my-charm: # This can be named anything you want
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A morsel for thought: we generally discourage the use of comments in bash, but I'm not sure where configs fit in, since we encourage users to copy them. As a general principle it's better to use the doc to explain things.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use comments pretty heavily in our config file templates. In this case, the goal of the comment is to counter something that's a fairly common misconception amongst charm authors (that the part name needs to be named the plugin name or the actual charm name) without having an aside about it, since that's out of scope for the document.

plugin: poetry
source: .

Ensure a recent enough version of ``pip`` and ``poetry``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Ensure a recent enough version of ``pip`` and ``poetry``
Select compatible versions of ``pip`` and ``poetry``

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``poetry`` plugin requires at least pip 22.3, released in October 2022. If the
charm's base uses an older version of pip, a newer version can be installed in the
build environment using a dependency part. Likewise, a charm may require a newer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newer

If I'm reading the code right, wouldn't it grab the latest?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default the plugin downloads from the distro repos. The example below gets the latest version.

version of poetry than is available in the distribution's repositories. The following
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
version of poetry than is available in the distribution's repositories. The following
version of Poetry than is available in the distribution's repositories. The following

``parts`` section can be used in place of the section above to upgrade pip and Poetry
for charms that build on Ubuntu 22.04 or earlier:

.. code-block:: yaml
:emphasize-lines: 2-9,11

parts:
poetry-deps:
plugin: nil
build-packages:
- curl
override-build: |
/usr/bin/python3 -m pip install pip==24.2
curl -sSL https://install.python-poetry.org | python3 -
ln -sf $HOME/.local/bin/poetry /usr/local/bin/poetry
my-charm: # This can be named anything you want
after: [poetry-deps]
plugin: poetry
source: .

Add optional dependency groups
------------------------------

If the charm has optional `dependency groups`_ that should be included when creating
the virtual environment, the ``poetry-with`` key can be used to include those groups
when creating the virtual environment.

.. note::
This is useful and encouraged, though not mandatory, for keeping track of
library dependencies (see below). For an example, see
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
library dependencies (see below). For an example, see
library dependencies, as covered in the next section. For an example, see

`postgresql-operator`_.

Include charm library dependencies
----------------------------------

Unlike the ``charm`` plugin, the ``poetry`` plugin does not install the dependencies
for included charmlibs. If any of the charm libraries used have PYDEPS, these will
need to be added to the charm's dependencies, potentially as their own
`dependency group <dependency groups>`_.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unspecified link.


To find these dependencies, check each library file for its ``PYDEPS``. A command
that can find these is::

find lib -name "*.py" -exec awk '/PYDEPS = \[/,/\]/' {} +

If run from the base directory of a charm, this will show all the PYDEPS declarations
from all loaded charm libs.

Include extra files
-------------------

A ``poetry`` plugin only includes the contents of the ``src`` and ``lib`` directories
as well as the generated virtual environment. If other files were previously included
from the main directory, they can be included again using the
:ref:`craft_parts_dump_plugin`:

.. code-block:: yaml
:emphasize-lines: 5-9

parts:
my-charm: # This can be named anything you want
plugin: poetry
source: .
version-file:
plugin: dump
source: .
stage:
- charm_version


.. _dependency groups: https://python-poetry.org/docs/managing-dependencies/#dependency-groups
.. _Poetry: https://python-poetry.org
.. _postgresql-operator: https://github.com/canonical/postgresql-operator/blob/3c7c783d61d4bee4ce64c190a9f7d4a78048e4e7/pyproject.toml#L22-L35
126 changes: 126 additions & 0 deletions docs/howto/charm-to-python.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
.. _howto-migrate-to-python:

Migrate from the ``charm`` plugin to the ``python`` plugin
==========================================================

The ``python`` Charmcraft plugin offers a faster, stricter means of packing an operator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Charmcraft plugin

Do we need to be this specific here? "Python Charmcraft plugin" tripped me up as a phrase.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because it works slightly differently from the Rockcraft plugin, the Snapcraft plugin, etc.

charm with a virtual environment. This guide shows how to migrate from a charm using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"packing with" is slightly ambiguous, so we might want to take special care with how we express this. If I'm understanding this correctly, could we say "packing a virtual environment inside an operator charm"?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The suggested language has implications that I'm not sure we want. The actual operator charm is their code - we're packing a charm package that includes the virtual environment.

the default ``charm`` plugin to using the ``python`` plugin.

Update ``charmcraft.yaml``
--------------------------

The first step is to update ``charmcraft.yaml`` to include the correct parts definition.
Depending on the history of a specific charm, it may not have an explicitly-included
``parts`` section determining how to build the charm. In this case, a ``parts`` section
can be created as follows:

.. code-block:: yaml

parts:
my-charm: # This can be named anything you want
plugin: python
source: .
python-requirements:
- requirements.txt # Or whatever your requirements file is called.

Ensure a recent enough version of ``pip``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Ensure a recent enough version of ``pip``
Select a compatible version of ``pip``

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``python`` plugin requires at least pip 22.3, released in October 2022. If the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pip 22.3, released in October 2022

Rather than mentioning the date, can we link to https://pypi.org/project/pip/22.3 instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can link there as well, but the date is also useful because it gives them a good idea as to whether their base distribution is going to have a new enough pip out of the box. For example, CentOS 7 and Ubuntu Jammy won't because they were all released before then (so they'll need to follow this), but Ubuntu Noble does. I don't want to list specific bases because that just gives us a treadmill of bases to update.

charm's base uses an older version of pip, a newer version can be installed in the
build environment using a dependency part. The following ``parts`` section can be
used in place of the section above to upgrade pip for charms that build on Ubuntu
22.04 or earlier:

.. code-block:: yaml
:emphasize-lines: 2-5,7

parts:
python-deps:
plugin: nil
override-build: |
/usr/bin/python3 -m pip install pip==24.2
my-charm: # This can be named anything you want
after: [python-deps]
plugin: python
source: .
python-requirements:
- requirements.txt # Or whatever your requirements file is called.

Flatten ``requirements.txt``
----------------------------

One difference between the ``python`` plugin and the ``charm`` plugin is that the
``python`` plugin does not install dependencies, so the ``requirements.txt`` file
must be a complete set of packages needed in the charm's virtual environment.

.. note::
There are several tools for creating an exhaustive ``requirements.txt`` file.
Charmcraft works with any as long as it generates a requirements file that ``pip``
understands. Because different versions of packages may have different
dependencies, it is recommended that the requirements file be generated using a
tool that will lock the dependencies to specific versions.
A few examples include:

- `uv export <https://docs.astral.sh/uv/reference/cli/#uv-export>`_
- `pip-compile <https://pip-tools.readthedocs.io/en/stable/cli/pip-compile/>`_
- `pip freeze <https://pip.pypa.io/en/stable/cli/pip_freeze/>`_

A basic ``requirements.txt`` file for a charm with no dependencies other than the
Operator framework may look something like::

ops==2.17.0
pyyaml==6.0.2
websocket-client==1.8.0

To check that the virtual environment for the charm would be valid, activate an
empty virtual environment and then run::

pip install --no-deps -r requirements.txt
pip check

Include charm library dependencies
----------------------------------

Unlike the ``charm`` plugin, the ``python`` plugin does not install the dependencies
for included charmlibs. If any of the charm libraries used have PYDEPS, these will
need to be added to a requirements file as well.

.. note::
All requirements files are included in the same ``pip`` command to prevent
conflicting requirements from overriding each other. However, this means
that a charm will fail to build if it has conflicting requirements. A single
``requirements.txt`` file, while not mandatory, is recommended.

To find these dependencies, check each library file for its ``PYDEPS``. A command
that can find these is::

find lib -name "*.py" -exec awk '/PYDEPS = \[/,/\]/' {} +

If run from the base directory of a charm, this will show all the PYDEPS declarations
from all loaded charm libs, which can be used to help generate the input for a tool
that generates ``requirements.txt``.

Include extra files
-------------------

A ``python`` plugin only includes the contents of the ``src`` and ``lib`` directories
as well as the generated virtual environment. If other files were previously included
from the main directory, they can be included again using the
:ref:`craft_parts_dump_plugin`:

.. code-block:: yaml
:emphasize-lines: 7-11

parts:
my-charm: # This can be named anything you want
plugin: python
source: .
python-requirements:
- requirements.txt # Or whatever your requirements file is called.
version-file:
plugin: dump
source: .
stage:
- charm_version
10 changes: 10 additions & 0 deletions docs/howto/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
.. _howto:

How-To
******

.. toctree::
:maxdepth: 2

charm-to-poetry
charm-to-python
11 changes: 11 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,20 @@ Most of Charmcraft's documentation is available there.
:maxdepth: 1
:hidden:

howto/index
reference/index
explanation/index

.. grid:: 1 1 2 2

.. grid-item-card:: `Tutorial <https://juju.is/docs/sdk/tutorials>`_

**Get started** with a hands-on introduction to Charmcraft

.. grid-item-card:: :ref:`How-to guides <howto>`

**Step-by-step guides** covering key operations and common tasks

.. grid:: 1 1 2 2
:reverse:

Expand Down
1 change: 0 additions & 1 deletion docs/reference/plugins/python_plugin.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@ During the build step, the plugin performs the following actions:
4. It copies any existing ``src`` and ``lib`` directories from your charm project into
the final charm.


Example
-------

Expand Down
Loading