diff --git a/staging_docs/reference/settings.md b/staging_docs/reference/settings.md new file mode 100644 index 00000000..d2a0cbb9 --- /dev/null +++ b/staging_docs/reference/settings.md @@ -0,0 +1,15 @@ +# Settings + +`pulp_python` adds configuration options to those offered by `pulpcore`. + +## PYTHON_GROUP_UPLOADS + +> This setting controls whether the uploading of packages through the PyPI APIs can be grouped +> inside http sessions. When enabled, Pulp tries to group the uploaded distributions of a package +> into one task generating one repository version. Defaults to False. + +## PYPI_API_HOSTNAME + +> This specifies the hostname where the PyPI API is served. It defaults to the fully qualified +> hostname of the system where the process is running. This needs to be adjusted if running behind +> a non local reverse proxy. diff --git a/staging_docs/user/guides/00-overview.md b/staging_docs/user/guides/00-overview.md new file mode 100644 index 00000000..b86b0cc7 --- /dev/null +++ b/staging_docs/user/guides/00-overview.md @@ -0,0 +1,59 @@ +# Overview + +The `` python` `` plugin extends [pulpcore](https://pypi.org/project/pulpcore/) to support +hosting python packages. This plugin is a part of the [Pulp Project](http://www.pulpproject.org), and assumes some familiarity with the [pulpcore documentation](https://docs.pulpproject.org/pulpcore/). + +If you are just getting started, we recommend getting to know the {doc}`basic +workflows`. + +The REST API documentation for `pulp_python` is available [here](restapi.html). + +## Features + +- `Create local mirrors of PyPI ` that you have full control over +- `Upload your own Python packages ` +- `Perform pip install ` from your Pulp Python repositories +- `Download packages on-demand ` to reduce disk usage +- Every operation creates a restorable snapshot with `Versioned Repositories ` +- Curate your Python repositories with allowed and disallowed packages +- Host content either [locally or on S3](https://docs.pulpproject.org/installation/storage.html) +- De-duplication of all saved content + +## Tech Preview + +Some additional features are being supplied as a tech preview. There is a possibility that +backwards incompatible changes will be introduced for these particular features. For a list of +features currently being supplied as tech previews only, see the {doc}`tech preview page +`. + +# How to use these docs + +The documentation here should be considered the **primary documentation for managing Python related content.** +All relevant workflows are covered here, with references to some pulpcore supplemental docs. +Users may also find pulpcore’s conceptual docs useful. + +This documentation falls into two main categories: + +> 1. `workflows-index` shows the **major features** of the Python plugin, with links to reference docs. +> 2. The [REST API Docs](restapi.html) are automatically generated and provide more detailed information for each +> minor feature, including all fields and options. + +## Table of Contents + +```{toctree} +:maxdepth: 1 + +installation +settings +workflows/index +changes +contributing +restapi/index +tech-preview +``` + +# Indices and tables + +- `genindex` +- `modindex` +- `search` diff --git a/staging_docs/user/guides/01-pypi.md b/staging_docs/user/guides/01-pypi.md new file mode 100644 index 00000000..78b7d91a --- /dev/null +++ b/staging_docs/user/guides/01-pypi.md @@ -0,0 +1,77 @@ +# Setup your own PyPI: + +This section guides you through the quickest way to setup `pulp_python` to act as your very own +private `PyPI`. + +## Create a Repository: + +Repositories are the base objects `Pulp` uses to store and organize its content. They are automatically +versioned when content is added or deleted and allow for easy rollbacks to previous versions. + +```bash +# Start by creating a new repository named "foo": +pulp python repository create --name foo +``` + +Repository Create Response: + +``` +{ + "pulp_href": "/pulp/api/v3/repositories/python/python/3fe0d204-217f-4250-8177-c83b30751fca/", + "pulp_created": "2021-06-02T14:54:53.387054Z", + "versions_href": "/pulp/api/v3/repositories/python/python/3fe0d204-217f-4250-8177-c83b30751fca/versions/", + "pulp_labels": {}, + "latest_version_href": "/pulp/api/v3/repositories/python/python/3fe0d204-217f-4250-8177-c83b30751fca/versions/1/", + "name": "foo", + "description": null, + "retained_versions": null, + "remote": null, + "autopublish": false + } +``` + +## Create a Distribution: + +Distributions serve the content stored in repositories so that it can be used by tools like `pip`. + +```bash +pulp python distribution create --name my-pypi --base-path my-pypi --repository foo +``` + +Distribution Create Response: + +``` +{ + "pulp_href": "/pulp/api/v3/distributions/python/pypi/e8438593-fd40-4654-8577-65398833c331/", + "pulp_created": "2021-06-03T20:04:18.233230Z", + "base_path": "my-pypi", + "base_url": "https://pulp3-source-fedora33.localhost.example.com/pypi/foo/", + "content_guard": null, + "pulp_labels": {}, + "name": "my-pypi", + "repository": "/pulp/api/v3/repositories/python/python/3fe0d204-217f-4250-8177-c83b30751fca/", + "publication": null, + "allow_uploads": true +} +``` + +## Upload and Install Packages: + +Packages can now be uploaded to the index using your favorite Python tool. The index url will be available +at `/pypi//simple/`. + +```bash +# Build custom package +python -m build $PLUGIN_SOURCE +# Upload built package distributions to my-pypi +twine upload --repository-url $BASE_ADDR/pypi/my-pypi/simple/ -u admin -p password "$PLUGIN_SOURCE"dist/* +``` + +Packages can then be installed using your favorite Python tool: + +```bash +pip install --trusted-host localhost -i $BASE_ADDR/pypi/my-pypi/simple/ shelf-reader +``` + +Now you have a fully operational Python package index. Check out the other workflows to see more features of +`pulp_python`. diff --git a/staging_docs/user/guides/02-sync.md b/staging_docs/user/guides/02-sync.md new file mode 100644 index 00000000..ba9cdcf5 --- /dev/null +++ b/staging_docs/user/guides/02-sync.md @@ -0,0 +1,193 @@ +# Synchronize a Repository + +Users can populate their repositories with content from an external source like PyPI by syncing +their repository. + + +## Create a Repository + +```bash +# Start by creating a new repository named "foo": +pulp python repository create --name foo +``` + +Repository Create Response: + +``` +{ + "pulp_href": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/", + "pulp_created": "2021-03-09T04:11:54.347921Z", + "versions_href": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/", + "pulp_labels": {}, + "latest_version_href": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/0/", + "name": "foo", + "description": null, + "remote": null + } +``` + +Reference (pulpcore): [Repository API Usage](https://docs.pulpproject.org/pulpcore/restapi.html#tag/Repositories) + + + +## Create a Remote + +Creating a remote object informs Pulp about an external content source. In this case, we will be +using a fixture, but Python remotes can be anything that implements the PyPI API. This can be PyPI +itself, a fixture, or even an instance of Pulp 2. + +```bash +# Create a remote that syncs some versions of shelf-reader into your repository. +pulp python remote create --name bar --url https://pypi.org/ --includes '["shelf-reader"]' +``` + +Remote Create Response: + +``` +{ + "pulp_href": "/pulp/api/v3/remotes/python/python/a9bb3a02-c7d2-4b2e-9b66-050a6c9b7cb3/", + "pulp_created": "2021-03-09T04:14:02.646835Z", + "name": "bar", + "url": "https://pypi.org/", + "ca_cert": null, + "client_cert": null, + "tls_validation": true, + "proxy_url": null, + "pulp_labels": {}, + "pulp_last_updated": "2021-03-09T04:14:02.646845Z", + "download_concurrency": 10, + "policy": "on_demand", + "total_timeout": null, + "connect_timeout": null, + "sock_connect_timeout": null, + "sock_read_timeout": null, + "headers": null, + "rate_limit": null, + "includes": [ + "shelf-reader" + ], + "excludes": [], + "prereleases": true, +} +``` + +Reference: [Python Remote Usage](../restapi.html#tag/Remotes:-Python) + +## A More Complex Remote + +If only the name of a project is specified, every distribution of every version of that project +will be synced. You can use the version_specifier field to ensure only distributions you care +about will be synced: + +```bash +pulp python remote create \ + --name 'complex-remote' \ + --url 'https://pypi.org/' \ + --includes '[ + "django~=2.0,!=2.0.1", + "pip-tools>=1.12,<=2.0", + "scipy", + "shelf-reader" + ]' +``` + +You can also use version specifiers to "exclude" certain versions of a project, like so: + +```bash +pulp python remote create \ + --name 'complex-remote' \ + --url 'https://pypi.org/' \ + --includes '[ + "django", + "scipy" + ]' \ + --excludes '[ + "django~=1.0", + "scipy" + ]' +``` + +You can also filter packages by their type, platform and amount synced through the "package_types", +"exclude_platforms", and "keep_latest_packages" fields respectively, like so: + +```bash +pulp python remote create \ + --name 'complex-filters' \ + --url 'https://pypi.org/' \ + --includes '["django"]' \ + --package-types '["sdist", "bdist-wheel"]' # only sync sdist and bdist-wheel package types \ + --exclude-platforms '["windows"]' # exclude any packages built for windows \ + --keep-latest-packages 5 # keep the five latest versions +``` + +Reference: [Python Remote Usage](../restapi.html#tag/Remotes:-Python) + + + +### Creating a remote to sync all of PyPI + +A remote can be setup to sync all of PyPI by not specifying any included packages like so: + +```bash +pulp python remote create \ + --name 'PyPI-mirror' \ + --url 'https://pypi.org/' \ + --excludes '[ + "django~=1.0", + "scipy" + ]' +``` + +By not setting the "includes" field Pulp will ask PyPI for all of its available packages to sync, minus the ones from +the excludes field. Default Python remotes are created with syncing policy "on_demand" because the most common +Python remotes involve syncing with PyPI which requires terabytes of disk space. This can be changed by +modifying the "policy" field. + +Syncing all of PyPI can take a long time depending on your network and disk speeds. Check out +`pull-through caching ` to learn about another way to mirror PyPI. + +## Sync repository foo with remote + +Use the remote object to kick off a synchronize task by specifying the repository to +sync with. You are telling pulp to fetch content from the remote and add to the repository. + +```bash +# Using the Remote we just created, we kick off a sync task +pulp python repository sync --name foo --remote bar + +# The sync command will by default wait for the sync to complete +# Use Ctrl+c or the -b option to send the task to the background + +# Show the latest version when sync is done +pulp python repository version show --repository foo +``` + +Repository Version Show Response (when complete): + +``` +{ + "pulp_href": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/1/", + "pulp_created": "2021-03-09T04:20:21.896132Z", + "number": 1, + "base_version": null, + "content_summary": { + "added": { + "python.python": { + "count": 2, + "href": "/pulp/api/v3/content/python/packages/?repository_version_added=/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/1/" + } + }, + "removed": {}, + "present": { + "python.python": { + "count": 2, + "href": "/pulp/api/v3/content/python/packages/?repository_version=/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/1/" + } + } + } +} +``` + +Reference: [Python Sync Usage](../restapi.html#operation/repositories_python_python_sync) + +Reference (pulpcore): [Repository Version Creation API Usage](https://docs.pulpproject.org/pulpcore/restapi.html#operation/repository_versions_list) diff --git a/staging_docs/user/guides/03-upload.md b/staging_docs/user/guides/03-upload.md new file mode 100644 index 00000000..b40d5187 --- /dev/null +++ b/staging_docs/user/guides/03-upload.md @@ -0,0 +1,112 @@ +# Upload and Manage Content + +Content can be added to a repository not only by synchronizing from a remote source but also by uploading the files directly into Pulp. + +## Create a repository + +If you don't already have a repository, create one. + +```bash +# Start by creating a new repository named "foo": +pulp python repository create --name foo +``` + +Repository GET response: + +``` +{ + "pulp_href": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/", + "pulp_created": "2021-03-09T04:11:54.347921Z", + "versions_href": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/", + "pulp_labels": {}, + "latest_version_href": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/0/", + "name": "foo", + "description": null, + "remote": null + } +``` + +## Upload a file to Pulp + +Each artifact in Pulp represents a file. They can be created during sync or created manually by uploading a file. + +```bash +# Get a Python package or choose your own +curl -O https://fixtures.pulpproject.org/python-pypi/packages/shelf-reader-0.1.tar.gz +export PKG="shelf-reader-0.1.tar.gz" + +# Upload it to Pulp +pulp python content upload --relative-path "$PKG" --file "$PKG" +``` + +Content Upload response: + +``` +{ + "pulp_href": "/pulp/api/v3/content/python/packages/f226894b-daa9-4152-9a04-595979ea5f9b/", + "pulp_created": "2021-03-09T04:47:13.066911Z", + "artifact": "/pulp/api/v3/artifacts/532b6318-add2-4208-ac1b-d6d37a39a97f/", + "filename": "shelf-reader-0.1.tar.gz", + "packagetype": "sdist", + "name": "shelf-reader", + "version": "0.1", + "metadata_version": "1.1", + "summary": "Make sure your collections are in call number order.", + "description": "too long to read" + "keywords": "", + "home_page": "https://github.com/asmacdo/shelf-reader", + "download_url": "", + "author": "Austin Macdonald", + "author_email": "asmacdo@gmail.com", + "maintainer": "", + "maintainer_email": "", + "license": "GNU GENERAL PUBLIC LICENSE Version 2, June 1991", + "requires_python": "", + "project_url": "", + "platform": "", + "supported_platform": "", + "requires_dist": "[]", + "provides_dist": "[]", + "obsoletes_dist": "[]", + "requires_external": "[]", + "classifiers": "[]" +} +``` + +## Add content to a repository + +Once there is a content unit, it can be added and removed from repositories using the add and remove commands + +```bash +# Add created PythonPackage content to repository +pulp python repository content add --repository foo --filename "shelf-reader-0.1.tar.gz" + +# After the task is complete, it gives us a new repository version +pulp python repository version show --repository foo +``` + +Repository Version Show Response (after task complete): + +``` +{ + "base_version": null, + "content_summary": { + "added": { + "python.python": { + "count": 1, + "href": "/pulp/api/v3/content/python/packages/?repository_version_added=/pulp/api/v3/repositories/python/python/931109d3-db86-4933-bf1d-45b4d4216d5d/versions/1/" + } + }, + "present": { + "python.python": { + "count": 1, + "href": "/pulp/api/v3/content/python/packages/?repository_version=/pulp/api/v3/repositories/python/python/931109d3-db86-4933-bf1d-45b4d4216d5d/versions/1/" + } + }, + "removed": {} + }, + "number": 1, + "pulp_created": "2020-05-28T21:04:54.403979Z", + "pulp_href": "/pulp/api/v3/repositories/python/python/931109d3-db86-4933-bf1d-45b4d4216d5d/versions/1/" +} +``` diff --git a/staging_docs/user/guides/04-publish.md b/staging_docs/user/guides/04-publish.md new file mode 100644 index 00000000..c14da59a --- /dev/null +++ b/staging_docs/user/guides/04-publish.md @@ -0,0 +1,130 @@ +# Publish and Host Your Python Content + +This section assumes that you have a repository with content in it. To do this, see the +[sync](02-sync.md) or [upload](03-upload.md) documentation. + +## Create a Publication (manually) + +Kick off a publish task by creating a new publication. The publish task will generate all the +metadata that `pip` needs to install packages (although it will need to be hosted through a +Distribution before it is consumable). + +```bash +# Create a new publication specifying the repository_version. +# Alternatively, you can specify just the repository, and Pulp will assume the latest version. +pulp python publication create --repository foo --version 1 + +# Publications can only be referenced through their pulp_href +PUBLICATION_HREF=$(pulp python publication list | jq -r .[0].pulp_href) +``` + +Publication Create Response: + +``` +{ + "pulp_href": "/pulp/api/v3/publications/python/pypi/cad6007d-7172-41d1-8c22-0ec95e1d242a/", + "pulp_created": "2021-03-09T04:30:16.686784Z", + "repository_version": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/versions/1/", + "repository": "/pulp/api/v3/repositories/python/python/8fbb24ee-dc91-44f4-a6ee-beec60aa542d/", + "distributions": [] +} +``` + +## Host a Publication (Create a Distribution) + +To host a publication, (which makes it consumable by `pip`), users create a distribution which +will serve the associated publication at `/pypi//` + +```bash +# Distributions are created asynchronously. Create one, and specify the publication that will +# be served at the base path specified. +pulp python distribution create --name foo --base-path foo --publication "$PUBLICATION_HREF" +``` + +Response: + +``` +{ + "pulp_href": "/pulp/api/v3/distributions/python/pypi/4839c056-6f2b-46b9-ac5f-88eb8a7739a5/", + "pulp_created": "2021-03-09T04:36:48.289737Z", + "base_path": "foo", + "base_url": "/pypi/foo/", + "content_guard": null, + "pulp_labels": {}, + "name": "foo", + "publication": "/pulp/api/v3/publications/python/pypi/a09111b1-6bce-43ac-aed7-2e8441c22704/" + } +``` + +## Automate Publication and Distribution + +With a little more initial setup, you can have publications and distributions for your repositories +updated automatically when new repository versions are created. + +```bash +# This configures the repository to produce new publications when a new version is created +pulp python repository update --name foo --autopublish +# This configures the distribution to be track the latest repository version for a given repository +pulp python distribution update --name foo --repository foo +``` + +!!! warning + Support for automatic publication and distribution is provided as a tech preview in Pulp 3. + Functionality may not work or may be incomplete. Also, backwards compatibility when upgrading + is not guaranteed. + + + + +## Enable Pull-Through Caching: + +Only packages present in your repository will be available from your index, but adding a remote source to +your distribution will enable the pull-through cache feature. This feature allows you to install any package +from the remote source and have Pulp store that package as orphaned content. + +```bash +# Add remote to distribution to enable pull-through caching +pulp python distribution update --name foo --remote bar +``` + +!!! warning + Support for pull-through caching is provided as a tech preview in Pulp 3. + Functionality may not work or may be incomplete. Also, backwards compatibility when upgrading + is not guaranteed. + + + + +## Use the newly created distribution + +The metadata and packages can now be retrieved from the distribution: + +```bash +$ http $BASE_ADDR/pypi/foo/simple/ +$ http $BASE_ADDR/pypi/foo/simple/shelf-reader/ +``` + +The content is also pip installable: + +```bash +$ pip install --trusted-host localhost -i $BASE_ADDR/pypi/foo/simple/ shelf-reader +``` + +If you don't want to specify the distribution path every time, you can modify your `pip.conf` +file. See the [pip docs](https://pip.pypa.io/en/stable/user_guide/#configuration) for more +detail.: + +```bash +$ cat pip.conf +``` + +``` +[global] +index-url = http://localhost:24817/pypi/foo/simple/ +``` + +The above configuration informs `pip` to install from `pulp`: + +```bash +$ pip install --trusted-host localhost shelf-reader +``` diff --git a/staging_docs/user/learn/tech-preview.md b/staging_docs/user/learn/tech-preview.md new file mode 100644 index 00000000..027fb932 --- /dev/null +++ b/staging_docs/user/learn/tech-preview.md @@ -0,0 +1,10 @@ +# Tech previews + +The following features are currently being released as part of a tech preview + +- New endpoint “pulp/api/v3/remotes/python/python/from_bandersnatch/” that allows for Python remote creation from a + Bandersnatch config file. +- PyPI’s json API at content endpoint ‘/pypi/\{package-name}/json’. Allows for basic Pulp-to-Pulp syncing. +- Fully mirror Python repositories provided PyPI and Pulp itself. +- `Twine` upload packages to indexes at endpoints '/simple\` or '/legacy'. +- Create pull-through caches of remote sources.