Update docs for "add_additional_resource" and other fixes (#233)

* ci: add ROOT v6.28 and Python v3.11 to test matrix

* docs: add "Adding resource links or files" etc.

* Add hepdata_lib.c_file_reader and hepdata_lib.helpers modules.
* Pin Sphinx<7 since docs fail to be built with Sphinx v7.
* Replace m2r2 with sphinx-mdinclude to avoid Mistune conflict.
* Replace deprecated "python setup.py test" with "pytest tests".
* Add section to "Developer information" about building docs.
* Add sections to "Usage" on "Adding resource links or files".
* Mention that venv can be used instead of virtualenv/virtualenvwrapper.

* examples: use add_additional_resource in notebooks

* hepdata_lib: add "type" to add_additional_resource

* Allow for type="HistFactory" to be passed as an additional argument.
* Extend test_submission.py to test the additional "type" argument.
* Add newlines to docstring of round_value_and_uncertainty for Sphinx.

* .gitignore: add files output by example notebooks

* .readthedocs.yml: install hepdata_lib package

* Installation needed to build the module index.
* Add "fail_on_warning: true" to avoid building broken docs.

* README.md: fix links to Jupyter notebooks

* Links to Jupyter notebooks previously broken on "Read the Docs".
* Binder links need to be given on a separate line to work properly.
* Add link to "Read the Docs" so it appears on main GitHub repo page.

* setup.py: add extras_require w/ test requirements

* Needed to install test dependencies with:  pip install -e ".[test]"
* Remove deprecated "setup_requires" and "tests_require".
* Add Python v3.11 to "Classifiers".

* ci: remove Python v3.11 from test matrix

* Python v3.11 incompatible with pylint v2.9.6 (installs wrapt v1.12.1).
* Add upper pin python_requires='>=3.6,<3.11' in setup.py.
* Use Python v3.10 in "Read the Docs" configuration file.

* docs: add "Analysing the code" section on Pylint

* Change "type" to "file_type" in argument of add_additional_resource.
* Disable some messages from Pylint when run on my local installation.

* hepdata_lib: add type of new "file_type" argument

* README.mb: fix Zenodo badge & give Python versions

* setup.py: remove upper bound on python_requires

* docs: reinstate SSH URL for "git clone"

* Also link to the GitHub docs for both HTTPS URLs and SSH URLs.

* ci: add Python 3.11 for tests but skip pylint step

* The pylint step can be added back in after #234 is resolved.

* Revert "ci: add Python 3.11 for tests but skip pylint step"

This reverts commit 77cc768.
The "Run pytest" step of the CI also fails with Python 3.11.

* ci/docs: rename master to main and improve docs

* Renamed 'master' branch to 'main' as with other HEPData repositories.
* Upgrade actions in GitHub Actions workflows to close Dependabot PRs.
* Revert to an SSH URL for main "git clone" with HTTPS URL in text.
* Remove recommendation to use virtualenv/virtualenvwrapper not 'venv'.

* pylint: remove 'disable-msg' and fix imports

* Remove 'disable-msg' comments introduced in commit dfa0a89.
* Add empty "__init__.py" file to tests/ directory.
* Add 'math' and 'array' modules to extension-pkg-whitelist in pylintrc.
* test_utilities.py needs to be a relative import after hepdata_lib.
* With these changes, pylint now gives 10.00/10 on my local setup.

* docs: tweak "Further examples" etc.

* Tweak "Further examples" so Binder on same line as text on GitHub.
* Remove pin on docutils==0.19, as not needed in recent Sphinx versions.
* Fix a copy-and-paste typo in setup.rst.

* docs: correct link to Python venv module

.github/workflows/docker.yml

@@ -21,23 +21,23 @@ jobs:
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
       - name: Log in to the Container registry
-        uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9
+        uses: docker/login-action@v3
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Extract metadata (tags, labels) for Docker
         id: meta
-        uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38
+        uses: docker/metadata-action@v5
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
 
       - name: Build and push Docker image
-        uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
+        uses: docker/build-push-action@v5
         with:
           context: .
           push: true

.github/workflows/release.yml

@@ -53,7 +53,7 @@ jobs:
         with:
           github_token: ${{ secrets.access_token }}
           tags: true
-          branch: master
+          branch: main
 
   # Action to build and publish distribution to PyPI and TestPyPI
   build-and-publish:
@@ -147,6 +147,6 @@ jobs:
 
     # Close issue
     - name: Close Issue
-      uses: peter-evans/close-issue@v1
+      uses: peter-evans/close-issue@v3
       with:
         issue-number: ${{ github.event.issue.number }}

.github/workflows/tests.yml

@@ -4,7 +4,7 @@ on:
   push:
   pull_request:
     branches:
-    - master
+    - main
   workflow_dispatch:
 
 # Needed for micromamba pickup
@@ -17,15 +17,15 @@ jobs:
 
     # Setup the Python that the ROOT binary was built against
     runs-on: ${{ matrix.os }}
-    # On push events run the CI only on master by default, but run on any branch if the commit message contains '[ci all]'
+    # On push events run the CI only on main by default, but run on any branch if the commit message contains '[ci all]'
     if: >-
       github.event_name != 'push'
-      || (github.event_name == 'push' && github.ref == 'refs/heads/master')
-      || (github.event_name == 'push' && github.ref != 'refs/heads/master' && contains(github.event.head_commit.message, '[ci all]'))
+      || (github.event_name == 'push' && github.ref == 'refs/heads/main')
+      || (github.event_name == 'push' && github.ref != 'refs/heads/main' && contains(github.event.head_commit.message, '[ci all]'))
     strategy:
       matrix:
         os: [ubuntu-latest]
-        root-version: ["6.24", "6.26"]
+        root-version: ["6.24", "6.26", "6.28"]
         python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
         exclude:
           - root-version: "6.24"
@@ -34,13 +34,17 @@ jobs:
             python-version: "3.6"
           - root-version: "6.26"
             python-version: "3.7"
+          - root-version: "6.28"
+            python-version: "3.6"
+          - root-version: "6.28"
+            python-version: "3.7"
         include:
           - os: macos-latest
-            root-version: "6.26"
+            root-version: "6.28"
             python-version: "3.10"
 
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
 
     - name: Setup Micromamba environment
       uses: mamba-org/provision-with-micromamba@v15

.gitignore

@@ -1,6 +1,8 @@
 # output from examples
 examples/submission.tar.gz
 examples/example_output/
+examples/output/
+examples/correlation.root
 
 # test output
 submission.tar.gz

.readthedocs.yml

@@ -5,19 +5,26 @@
 # Required
 version: 2
 
-# Build documentation in the docs/ directory with Sphinx
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
+# Build documentation in the "docs/" directory with Sphinx
 sphinx:
   configuration: docs/conf.py
-
-# Build documentation with MkDocs
-#mkdocs:
-#  configuration: mkdocs.yml
+  # Fail on all warnings to avoid broken references
+  fail_on_warning: true
 
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: all
 
-# Optionally set the version of Python and requirements required to build your docs
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
 python:
-  version: 3.7
   install:
-    - requirements: docs/requirements.txt
+  - requirements: docs/requirements.txt
+  - method: pip
+    path: .

README.md

@@ -1,14 +1,18 @@
 # hepdata_lib
 
-[![DOI](https://zenodo.org/badge/129248575.svg)](https://zenodo.org/badge/latestdoi/129248575)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1217998.svg)](https://doi.org/10.5281/zenodo.1217998)
 [![PyPI version](https://badge.fury.io/py/hepdata-lib.svg)](https://badge.fury.io/py/hepdata-lib)
 [![Actions Status](https://github.com/HEPData/hepdata_lib/workflows/tests/badge.svg)](https://github.com/HEPData/hepdata_lib/actions)
-[![Coverage Status](https://codecov.io/gh/HEPData/hepdata_lib/graph/badge.svg?branch=master)](https://codecov.io/gh/HEPData/hepdata_lib?branch=master)
+[![Coverage Status](https://codecov.io/gh/HEPData/hepdata_lib/graph/badge.svg?branch=main)](https://codecov.io/gh/HEPData/hepdata_lib?branch=main)
 [![Documentation Status](https://readthedocs.org/projects/hepdata-lib/badge/)](http://hepdata-lib.readthedocs.io/)
 [![Docker image](https://github.com/HEPData/hepdata_lib/actions/workflows/docker.yml/badge.svg)](https://github.com/HEPData/hepdata_lib/pkgs/container/hepdata_lib)
 
 Library for getting your data into HEPData
 
+- Documentation: https://hepdata-lib.readthedocs.io
+
+This code works with Python 3.6, 3.7, 3.8, 3.9 or 3.10.
+
 ## Installation
 
 It is highly recommended you install `hepdata_lib` into a [virtual environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).
@@ -23,7 +27,7 @@ If you are not sure about your Python environment, please also see below how to
 
 For using `hepdata_lib`, you don't even need to install it, but can use the [binder](https://mybinder.org/) or [SWAN](https://swan.cern.ch/) (CERN-only) services using one of the buttons below and following the instructions in the notebook with name [Getting_started](examples/Getting_started.ipynb):
 
-[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/master?filepath=examples/Getting_started.ipynb)
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/main?filepath=examples/Getting_started.ipynb)
 [![SWAN](https://swanserver.web.cern.ch/swanserver/images/badge_swan_white_150.png)](https://cern.ch/swanserver/cgi-bin/go/?projurl=https://github.com/HEPData/hepdata_lib.git)
 
 You can also use the Docker image:
@@ -53,11 +57,21 @@ Unpacking the image can take a few minutes the first time you use it. Please be
 
 There are a few more examples available that can directly be run using the [binder](https://mybinder.org/) links below or using [SWAN](https://swan.cern.ch/) (CERN-only, please use LCG release LCG_94 or later) and selecting the corresponding notebook manually:
 
-- [Reading in text files](examples/Getting_started.ipynb) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/master?filepath=examples/Getting_started.ipynb)
-- [Reading in a CMS combine ntuple](examples/combine_limits.ipynb) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/master?filepath=examples/combine_limits.ipynb)
-- [Reading in ROOT histograms](examples/reading_histograms.ipynb) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/master?filepath=examples/reading_histograms.ipynb)
-- [Reading a correlation matrix](examples/correlation.ipynb) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/master?filepath=examples/correlation.ipynb)
-- [Reading TGraph and TGraphError from '.C' files](examples/read_c_file.ipynb) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/master?filepath=examples/read_c_file.ipynb)
+- [Reading in text files](https://github.com/HEPData/hepdata_lib/blob/main/examples/Getting_started.ipynb)
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/main?filepath=examples/Getting_started.ipynb)
+<br/><br/>
+- [Reading in a CMS combine ntuple](https://github.com/HEPData/hepdata_lib/blob/main/examples/combine_limits.ipynb)
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/main?filepath=examples/combine_limits.ipynb)
+<br/><br/>
+- [Reading in ROOT histograms](https://github.com/HEPData/hepdata_lib/blob/main/examples/reading_histograms.ipynb)
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/main?filepath=examples/reading_histograms.ipynb)
+<br/><br/>
+- [Reading a correlation matrix](https://github.com/HEPData/hepdata_lib/blob/main/examples/correlation.ipynb)
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/main?filepath=examples/correlation.ipynb)
+<br/><br/>
+- [Reading TGraph and TGraphError from '.C' files](https://github.com/HEPData/hepdata_lib/blob/main/examples/read_c_file.ipynb)
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/HEPData/hepdata_lib/main?filepath=examples/read_c_file.ipynb)
+<br/><br/>
 
 ## External dependencies
 

docs/Makefile

@@ -2,7 +2,7 @@
 #
 
 # You can set these variables from the command line.
-SPHINXOPTS    =
+SPHINXOPTS    = -W --keep-going
 SPHINXBUILD   = sphinx-build
 SPHINXPROJ    = hepdata_lib
 SOURCEDIR     = .

docs/conf.py

@@ -35,7 +35,7 @@ def __getattr__(cls, name):
 # -- Project information -----------------------------------------------------
 
 project = 'hepdata_lib'
-copyright = '2018-2022, Andreas Albert, Clemens Lange'
+copyright = '2018-2023, Andreas Albert, Clemens Lange'
 author = 'Andreas Albert, Clemens Lange'
 
 # The short X.Y version
@@ -55,7 +55,7 @@ def __getattr__(cls, name):
 # ones.
 extensions = [
     'sphinx.ext.githubpages',
-    'm2r2',
+    'sphinx_mdinclude',
     'sphinx.ext.autodoc'
 ]
 
@@ -76,7 +76,7 @@ def __getattr__(cls, name):
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -103,7 +103,7 @@ def __getattr__(cls, name):
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+# html_static_path = ['_static']
 
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.

docs/contributing.rst

@@ -13,7 +13,7 @@ Please see :ref:`sec-setup-developers` on how to set up the library for developm
 Using bumpversion
 -----------------------------
 
-bumpversion_ allows to update the library version consistently over all files. Please do not change the version manually, but use the following steps instead after a pull request has been merged into the ``master`` branch. Depending on the amount of changes, choose accordingly from:
+bumpversion_ allows to update the library version consistently over all files. Please do not change the version manually, but use the following steps instead after a pull request has been merged into the ``main`` branch. Depending on the amount of changes, choose accordingly from:
 
 - ``patch`` = ``+0.0.1``
 - ``minor`` = ``+0.1.0``
@@ -24,17 +24,17 @@ Execute the following commands:
 ::
 
     pip install --upgrade bumpversion
-    git checkout master
+    git checkout main
     git pull
     bumpversion patch # adjust accordingly
-    git push origin master --tags
+    git push origin main --tags
 
 The files in which the versions are updated as well as the current version can be found in the `.bumpversion.cfg`_. You need appropriate rights for the repository to be able to push the tag.
 
 .. _sec-dev-pypi:
 
 Uploading to PyPI
------------------------------
+-----------------
 
 Once a new version has been tagged, the package should be uploaded to the Python Package Index (PyPI_).
 For the markdown formatting to work, ``twine>=1.11.0`` is required.
@@ -65,14 +65,14 @@ You should then find the new version at `this location`_. You need to be a maint
 
 
 .. _bumpversion: https://github.com/peritus/bumpversion
-.. _.bumpversion.cfg: https://github.com/HEPData/hepdata_lib/blob/master/.bumpversion.cfg
+.. _.bumpversion.cfg: https://github.com/HEPData/hepdata_lib/blob/main/.bumpversion.cfg
 .. _PyPI: https://pypi.org
 .. _PyPI test server: https://test.pypi.org/project/hepdata_lib/
 .. _this location: https://pypi.org/project/hepdata_lib/
 .. _python packaging documentation: https://packaging.python.org/tutorials/packaging-projects/
 
 Creating a new release automatically via an issue
------------------------------
+-------------------------------------------------
 
 Opening an new issue automatically creates a new release if:
 

docs/dev.rst

@@ -1,21 +1,22 @@
 Developer information
-======================
+=====================
 
 The testing system
---------------------
+------------------
 
 While further developing the code base, we want to ensure that any changes we make do not accidentally break existing functionality. For this purpose, we use continuous integration via GitHub Actions_. Practically, this means that we define a set of test code that is automatically run in a predefined environment every time we push to a branch or create a pull request. If the test code runs successfully, we know that everything works as expected.
 
-To run the tests, move into the ``hepdata_lib`` directory and run
+To run the tests, move into the ``hepdata_lib`` directory while in your virtual environment and run
 
 ::
 
-    python setup.py test
+    pip install -e ".[test]"
+    pytest tests
 
 It is a good idea to run the tests manually to ensure that your changes do not cause any issues.
 
 Definition of test cases
-+++++++++++++++++++++++++
+++++++++++++++++++++++++
 
 Test cases are defined in the ``test`` subfolder of the repository. Independent sets of test code are defined in individual files in that directory, with each files being named like ``test_*.py``.
 
@@ -40,12 +41,12 @@ If all functions run without raising exceptions, the test is considered to be pa
 
 
 When to test
-~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~
 
 Tests should be added any time functionality is added. If functionality is modified, so should the tests.
 
 What to test
-~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~
 
 Generally, the tests should ensure that the code behaves as expected, whatever that may mean. They should be sufficiently rigorous to make sure that nothing can break silently, and that outputs are correct.
 
@@ -57,7 +58,7 @@ Broad inspiration for aspects to check:
 
 
 How to test
-~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~
 
 The ``TestCase`` base class provides functionality to help implement simple checks to verify the behavior. A test can immediately be made to fail by calling ``self.fail()``. For convenience, additional functions like ``assertTrue``, ``assertFalse``, etc. are provided, which work like normal python assertions. If the assertion fails, the test is considered to be failed. Additionally, the ``assertRaises`` method can be used as a context to ensure that exceptions are raised as expected. Here's a simple example:
 
@@ -95,10 +96,28 @@ Note that this is an overly simple example case: In a real testing case, you sho
 
 Check out the unittest package documentation_ for details of the available testing functionality.
 
+.. _Actions: https://docs.github.com/en/actions
+.. _documentation: https://docs.python.org/2/library/unittest.html#unittest.TestCase
+
 
+Building the documentation
+--------------------------
 
+After installing the ``hepdata_lib`` package, move into the ``hepdata_lib/docs`` directory and install the additional necessary packages into your virtual environment:
 
+::
 
+    pip install -r requirements.txt
 
-.. _Actions: https://docs.github.com/en/actions
-.. _documentation: https://docs.python.org/2/library/unittest.html#unittest.TestCase
+Then you can build the documentation locally with Sphinx using ``make html`` and view the output by opening a web browser at ``_build/html/index.html``.
+
+
+Analysing the code
+------------------
+
+::
+
+    pylint hepdata_lib/*.py
+    pylint tests/*.py --rcfile=tests/pylintrc
+
+These commands are run by GitHub Actions, so you should first check locally that no issues are flagged.

docs/requirements.txt

@@ -1,4 +1,3 @@
-m2r2==0.3.3
+Sphinx<7
 sphinx_rtd_theme
-docutils==0.19
-mistune<2.0.0
+sphinx-mdinclude