From 8db01419d3d956f95b8e3e4e150fdf686dfc9d41 Mon Sep 17 00:00:00 2001 From: Douglas Cerna Date: Thu, 7 Dec 2023 09:57:52 -0600 Subject: [PATCH] Lint Markdown files --- .markdownlint.jsonc | 10 ++++ .pre-commit-config.yaml | 16 +++++++ CONTRIBUTING.md | 77 ++++++++++++++++++------------ README.md | 103 +++++++++++++++++++++++++--------------- docs/fixity.1.md | 27 +++++------ 5 files changed, 152 insertions(+), 81 deletions(-) create mode 100644 .markdownlint.jsonc diff --git a/.markdownlint.jsonc b/.markdownlint.jsonc new file mode 100644 index 0000000..dc8f15e --- /dev/null +++ b/.markdownlint.jsonc @@ -0,0 +1,10 @@ +{ + "default": true, + "MD010": { + "code_blocks": false + }, + "MD013": { + "code_blocks": false, + "tables": false + } +} diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 07bab0c..dfdef45 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -21,3 +21,19 @@ repos: additional_dependencies: - flake8-bugbear==23.9.16 - flake8-comprehensions==3.14.0 +- repo: https://github.com/igorshubovych/markdownlint-cli + rev: v0.37.0 + hooks: + - id: markdownlint + exclude: | + (?x)^( + ISSUE_TEMPLATE\.md + ) +- repo: https://github.com/thlorenz/doctoc + rev: v2.2.0 + hooks: + - id: doctoc + files: | + (?x)^( + (CONTRIBUTING|README)\.md + ) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 52d3cd4..d80c6a3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,13 +12,15 @@ bug reports, answering questions on the [mailing list](https://groups.google.com/forum/#!forum/archivematica), or writing code which can be incorporated into Archivematica itself. +## Table of contents + -**Table of Contents** - [Submitting bugs](#submitting-bugs) -- [Submitting enhancement ideas](#submitting-enhancements) +- [Submitting enhancement ideas](#submitting-enhancement-ideas) - [Submitting code changes](#submitting-code-changes) + - [Permalinks](#permalinks) - [Getting started](#getting-started) - [When to submit code for review?](#when-to-submit-code-for-review) - [Opening the pull request](#opening-the-pull-request) @@ -37,15 +39,17 @@ which can be incorporated into Archivematica itself. - [Commit History](#commit-history) - [Commits should be specific and atomic](#commits-should-be-specific-and-atomic) - [Every commit should work](#every-commit-should-work) + - [Commit messages](#commit-messages) - [Commit summaries should be concise](#commit-summaries-should-be-concise) - [Commit messages should be as detailed as they need to be (and no more)](#commit-messages-should-be-as-detailed-as-they-need-to-be-and-no-more) + - [More information on commit messages](#more-information-on-commit-messages) ## Submitting bugs If you find a security vulnerability, do NOT open an issue. Email -security@artefactual.com instead. + instead. All issues about Archivematica or related libraries should be filed in the [Archivematica Issues repository](https://github.com/archivematica/Issues). @@ -57,14 +61,14 @@ local problem! Useful questions to answer if you're having problems include: -* What version of Archivematica and the Storage Service are you using? -* How was Archivematica installed? Package install, Ansible, etc? -* Was this a fresh install or an upgrade? -* What did you do to cause this bug to happen? -* What did you expect to happen? -* What did you see instead? -* Can you reproduce this reliably? -* If a specific Job is failing, what output did it produce? This is available +- What version of Archivematica and the Storage Service are you using? +- How was Archivematica installed? Package install, Ansible, etc? +- Was this a fresh install or an upgrade? +- What did you do to cause this bug to happen? +- What did you expect to happen? +- What did you see instead? +- Can you reproduce this reliably? +- If a specific Job is failing, what output did it produce? This is available by clicking on the gear icon. ## Submitting enhancement ideas @@ -182,7 +186,7 @@ The Archivematica contributor's agreement is based almost verbatim on the [contributor license](http://www.apache.org/licenses/icla.txt). If you have any questions or concerns about the Contributor's Agreement, -please email us at agreement@artefactual.com to discuss them. +please email us at to discuss them. ### Why do I have to sign a Contributor's Agreement? @@ -209,14 +213,16 @@ be, rather than fighting legal battles over contributions. Please print out, read, sign, and scan the [contributor agreement](https://wiki.archivematica.org/images/2/25/Contributor_agreement.txt) -and email it to agreement@artefactual.com +and email it to Alternatively, you may send an original signed agreement to: - Artefactual Systems Inc. - 201 - 301 Sixth Street - New Westminster BC V3L 3A7 - Canada +```text +Artefactual Systems Inc. +201 - 301 Sixth Street +New Westminster BC V3L 3A7 +Canada +``` ## Contribution standards @@ -227,11 +233,11 @@ community style guidelines. Newly-written code should conform to PEP-8 style. PEP8 is a daunting document, but there are very good linters available that you can run to check style in your code. -* The [Black](https://github.com/ambv/black) tool formats the code +- The [Black](https://github.com/ambv/black) tool formats the code automatically. The output is deterministic for any given input. Editor integration is possible. -* The [flake8](https://pypi.python.org/pypi/flake8) tool checks for style +- The [flake8](https://pypi.python.org/pypi/flake8) tool checks for style problems as well as errors and complexity. It can be used at the command line or as a plugin in your preferred text editor/IDE. The Archivematica [continuous integration system](https://travis-ci.org/artefactual/archivematica) @@ -248,10 +254,10 @@ code-review. A few additional stylistic preferences might not get flagged by linters: -* Don't use variable or parameter names that shadow builtin functions and +- Don't use variable or parameter names that shadow builtin functions and types. For example, don't name a variable "file". (Unfortunately, Python uses many useful names for its builtin types and functions.) -* Sort imports alphabetically within their grouping to reduce duplicate +- Sort imports alphabetically within their grouping to reduce duplicate imports. #### Exceptions @@ -272,10 +278,10 @@ however, if possible, it's great if your new code does conform. When working in sections of code that don't conform to PEP8, it's okay to relax a few PEP8 rules in order to match existing code. In particular: -* When modifying an existing function which uses camelCase variables or +- When modifying an existing function which uses camelCase variables or parameters, it's okay to make your new variables/parameters camelCase to match. -* When adding new functions to a module or class that uses camelCase naming, +- When adding new functions to a module or class that uses camelCase naming, it's okay to make your new function and its parameters camelCase to match. Try to use snake_case internally, however. @@ -293,8 +299,8 @@ Generally any function which isn't obvious (any function longer than a line or two) should have a docstring. When in doubt: document! Python's [PEP 257](https://www.python.org/dev/peps/pep-0257/) document provides a useful guideline for docstring style. Generally, prefer using -[Sphinx-compatible docstrings](http://pythonhosted.org/an_example_pypi_project/sphinx.html#function-definitions). More -[examples](http://sphinx-doc.org/domains.html#info-field-lists) and +[Sphinx-compatible docstrings](http://pythonhosted.org/an_example_pypi_project/sphinx.html#function-definitions). +More [examples](http://sphinx-doc.org/domains.html#info-field-lists) and [attributes to use](http://sphinx-doc.org/domains.html#the-python-domain) can be found on the Sphinx website. @@ -363,13 +369,19 @@ and b) what the change is. For example: Clear commit summary: -> Replace 404 messages with a user-friendly one +```text +Replace 404 messages with a user-friendly one +``` Unclear commit summaries: -> Fixed some normalization bugs +```text +Fixed some normalization bugs +``` -> Bugfixes +```text +Bugfixes +``` The unclear messages make it hard to tell at a glance what changed, and that makes browsing the commit history harder. @@ -378,7 +390,9 @@ A commit message should use the [imperative mood](https://chris.beams.io/posts/git-commit/#imperative) which should always be able to complete the following sentence: - If applied, this commit will +```text +If applied, this commit will +``` #### Commit messages should be as detailed as they need to be (and no more) @@ -396,7 +410,8 @@ made. Make their life easier. Each line of a commit message should be no more than 72 characters in width. The following is an outline of a commit message combined with an ideal commit summary: -``` + +```text Capitalized, short (50 chars or less) summary More detailed explanatory text, if necessary. Wrap it to about 80 @@ -408,7 +423,9 @@ two together. Further content comes after a blank line. ``` + ##### More information on commit messages + For more information about how to write a good commit message and to understand some of the rationale behind our approach, please see this article from Chris Beams on diff --git a/README.md b/README.md index b540787..75f5051 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,31 @@ +# fixity + +[![PyPI version](https://img.shields.io/pypi/v/fixity.svg)](https://pypi.python.org/pypi/fixity) [![GitHub CI](https://github.com/artefactual/fixity/actions/workflows/test.yml/badge.svg)](https://github.com/artefactual/fixity/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/artefactual/fixity/branch/master/graph/badge.svg?token=wiga5iF7CK)](https://codecov.io/gh/artefactual/fixity) -fixity -====== +## Table of contents + + + -* [About](#about) -* [Check fixity](#check-fixity) -* [Fixity errors](#fixity-errors) - * [Information has been deleted from a file](#information-has-been-deleted-from-a-file) - * [A character in a file has been modified](#a-character-in-a-file-has-been-modified) - * [A file has been removed from the package](#a-file-has-been-removed-from-the-package) - * [A file has been added to the package](#a-file-has-been-added-to-the-package) - * [A manifest has been removed from the package](#a-manifest-has-been-removed-from-the-package) - * [How this looks in the Storage Service](#how-this-looks-in-the-storage-service) -* [Installation](#installation) -* [Usage](#usage) -* [Security](#security) -* [Copyright](#copyright) +- [About](#about) +- [Check fixity](#check-fixity) +- [Fixity errors](#fixity-errors) + - [Information has been deleted from a file](#information-has-been-deleted-from-a-file) + - [A character in a file has been modified](#a-character-in-a-file-has-been-modified) + - [A file has been removed from the package](#a-file-has-been-removed-from-the-package) + - [A file has been added to the package](#a-file-has-been-added-to-the-package) + - [A manifest has been removed from the package](#a-manifest-has-been-removed-from-the-package) + - [How this looks in the Storage Service](#how-this-looks-in-the-storage-service) +- [Installation](#installation) +- [Usage](#usage) +- [Security](#security) +- [Copyright](#copyright) + -About ------ +## About Fixity is a command line tool that assists in checking fixity for AIPs stored in the Archivematica Storage Service. @@ -36,8 +41,7 @@ Fixity is a client application that calls the Storage Service's [Check Fixity](https://wiki.archivematica.org/Storage_Service_API#Check_fixity) API endpoint for a single AIP or for each AIP in the Storage Service. -Check fixity ------------- +## Check fixity The Storage Service's Check Fixity endpoint can be used to trigger a fixity check on an individual package. When the Fixity client calls this endpoint, the @@ -53,7 +57,8 @@ The URL is constructed as follows: And we can supply authentication information in a HTTP request using a command line tool such as [HTTPie](https://httpie.org): -```bash + +```shell http -v --pretty=format \ GET "http://127.0.0.1:62081/api/v2/file/5e21dd0d-190e-4ffb-b752-76d860bea898/check_fixity/" \ Authorization:"ApiKey test:test" @@ -61,6 +66,7 @@ http -v --pretty=format \ This results in the following JSON output showing us that the Fixity check was a 'success', that is, there were no errors: + ```json { "failures": { @@ -80,7 +86,9 @@ This detailed JSON output is recorded in Fixity's internal database. Summary information about errors is printed to the console. The output for package `5e21dd0d-190e-4ffb-b752-76d860bea898` would simply look as follows: -`Fixity scan succeeded for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898` +```console +Fixity scan succeeded for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 +``` The Check Fixity endpoint updates the Storage Service `fixitylog` model with details of the last check completed. @@ -90,6 +98,7 @@ Storage Service Packages panel by going to `{storage-service-url}/packages/` Administrators of the Storage Service can query the Storage Service database as follows: + ```sql select package_id as "AIP UUID", @@ -103,6 +112,7 @@ limit 1; ``` And we can see the details of the check in the SQL output: + ```sql +--------------------------------------+----------------------------+---------+---------------+ | AIP UUID | Fixity Last Checked | success | error_details | @@ -111,20 +121,21 @@ And we can see the details of the check in the SQL output: +--------------------------------------+----------------------------+---------+---------------+ ``` -Fixity errors -------------- +## Fixity errors The Storage Service can detect multiple categories of error. Some are shown as outputs of Fixity below with the corresponding detailed information that can be accessed from Fixity's database: ### Information has been deleted from a file -``` + +```console Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Oxum error. Found 10 files and 126404 bytes on disk; expected 10 files and 126405 bytes.) ``` **Detail:** + ```json { "failures": { @@ -143,11 +154,13 @@ Found 10 files and 126404 bytes on disk; expected 10 files and 126405 bytes.) ``` ### A character in a file has been modified -``` + +```console Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (invalid bag) ``` **Detail:** + ```json { "failures": { @@ -174,12 +187,14 @@ Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (invalid bag) ``` ### A file has been removed from the package -``` + +```console Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Oxum error. Found 9 files and 126386 bytes on disk; expected 10 files and 126405 bytes.) ``` **Detail:** + ```json { "failures": { @@ -198,12 +213,14 @@ Found 9 files and 126386 bytes on disk; expected 10 files and 126405 bytes.) ``` ### A file has been added to the package -``` + +```console Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Oxum error. Found 11 files and 126405 bytes on disk; expected 10 files and 126405 bytes.) ``` **Detail:** + ```json { "failures": { @@ -223,12 +240,14 @@ Found 11 files and 126405 bytes on disk; expected 10 files and 126405 bytes.) ``` ### A manifest has been removed from the package -``` + +```console Fixity scan failed for AIP: 5e21dd0d-190e-4ffb-b752-76d860bea898 (Missing manifest file) ``` **Detail:** + ```json { "failures": { @@ -250,6 +269,7 @@ manifest file) For any error in the Fixity check the Storage Service database maintains a summary log, see for example when data in a file has been modified: + ```sql +--------------------------------------+----------------------------+---------+---------------+ | AIP UUID | Fixity Last Checked | success | error_details | @@ -258,24 +278,26 @@ summary log, see for example when data in a file has been modified: +--------------------------------------+----------------------------+---------+---------------+ ``` -Installation ------------- +## Installation Installation of Fixity can be completed with the following steps: 1. Checkout or link the code to `/usr/lib/archivematica/fixity` 1. Go to `/usr/lib/archivematica/` + ```bash user@root:~$ cd /usr/lib/archivematica/ ``` 2. Clone the code: + ```bash user@root:/usr/lib/archivematica/$ git clone https://github.com/artefactual/fixity.git ``` Once this is complete, the directory `/usr/lib/archivematica/fixity` should exist. `cd` back to the home directory + ```bash user@root:/usr/lib/archivematica/$ cd ``` @@ -284,11 +306,13 @@ Installation of Fixity can be completed with the following steps: dependencies in it 1. Switch to root + ```bash user@root:~$ sudo -i ``` 2. Run: + ```bash root@host:~# virtualenv /usr/share/python/fixity root@host:~# source /usr/share/python/fixity/bin/activate @@ -299,6 +323,7 @@ Installation of Fixity can be completed with the following steps: 3. Create a symlink from the executable to `/usr/local/bin`. You must still be root. + ```bash (fixity)root@host:/usr/lib/archivematica/fixity# ln -s /usr/share/python/fixity/bin/fixity /usr/local/bin/fixity ``` @@ -307,6 +332,7 @@ Installation of Fixity can be completed with the following steps: `/etc/profile.d/fixity.sh` is recommended: 1. To create the file: + ```bash (fixity)root@host:/usr/lib/archivematica/fixity# touch /etc/profile.d/fixity.sh (fixity)root@host:/usr/lib/archivematica/fixity# nano /etc/profile.d/fixity.sh @@ -315,6 +341,7 @@ Installation of Fixity can be completed with the following steps: 2. You are now editing the environment variables file. You should use the URL of your Storage Service, and the username and API key of one Storage Service user. Replace the URL, user and key with your data. + ```bash #!/bin/bash export STORAGE_SERVICE_URL=http://localhost:8000 @@ -324,6 +351,7 @@ Installation of Fixity can be completed with the following steps: 3. Optionally, if you are using Fixity with a reporting service, you can also add: + ```bash export REPORT_URL=http://myurl.com export REPORT_USERNAME=myuser @@ -331,23 +359,27 @@ Installation of Fixity can be completed with the following steps: ``` 4. Load the variables from the file. + ```bash (fixity)root@host:/usr/lib/archivematica/fixity# source /etc/profile.d/fixity.sh ``` 5. Run the tool with sudo or as root the first time. Subsequent runs can be with any user. + ```bash (fixity)root@host:/usr/lib/archivematica/fixity# fixity scanall ``` 6. To exit the virtualenv: + ```bash (fixity)root@host:/usr/lib/archivematica/fixity# deactivate root@host:/usr/lib/archivematica/fixity# ``` And to exit the root user: + ```bash root@host:/usr/lib/archivematica/fixity# exit user@host:~$ @@ -355,26 +387,23 @@ Installation of Fixity can be completed with the following steps: 7. After the initial install, to run fixity you only need to load the variables you defined earlier and run fixity. + ```bash user@host:~$ source /etc/profile.d/fixity.sh user@host:~$ fixity scanall ``` -Usage ------ +## Usage For more information on usage, consult the [manpage](docs/fixity.1.md). -Security --------- +## Security If you have a security concern about Archivematica or any of its companion repositories, please see the [Archivematica security policy](https://github.com/artefactual/archivematica/security/policy) for information on how to safely report a vulnerability. - -Copyright ---------- +## Copyright Fixity is copyright 2014-2018 Artefactual Systems Inc. diff --git a/docs/fixity.1.md b/docs/fixity.1.md index 56f8230..cfccb8b 100644 --- a/docs/fixity.1.md +++ b/docs/fixity.1.md @@ -1,5 +1,4 @@ -fixity(1) -- Check fixity for AIPs -================================== +# fixity(1) -- Check fixity for AIPs ## SYNOPSIS @@ -24,28 +23,28 @@ running; see the section on _ENVIRONMENT VARIABLES_ for information. ## OPTIONS - * `--throttle` : +* `--throttle `: Time (in seconds) to wait when scanning multiple AIPs. This can help reduce extended disk load on the Storage Service filesystem on which the AIPs reside. - * `--force-local`: +* `--force-local`: Request the Storage Service performs a local fixity check, instead of using the Space's fixity (this is only available for Arkivum Spaces). - * `--debug`: +* `--debug`: Print extra debugging output. ## COMMANDS - * `scan` : +* `scan `: Run a fixity scan on a single AIP, using the specified UUID. If the UUID is malformed, or the Storage Service does not have an AIP with the specified UUID, this will produce an error and exit 1. After the scan completes, a brief report will be printed with information on whether the scan succeeded or failed. - * `scanall`: +* `scanall`: Run a fixity scan on every AIP registered with the target Storage Service instance. This command does not take any arguments. A brief report will be printed after every AIP is scanned. @@ -58,26 +57,26 @@ running; see the section on _ENVIRONMENT VARIABLES_ for information. The following environment variables **must** be exported in the environment for fixity to operate. - * **STORAGE_SERVICE_URL**: +* **STORAGE_SERVICE_URL**: The base URL to the storage service instance to scan. Must include the port number for non port 80 installations. Example: - http://localhost:8000/ + - * **STORAGE_SERVICE_USER**: +* **STORAGE_SERVICE_USER**: Username for API authentication with the storage service. Example: test - * **STORAGE_SERVICE_KEY**: +* **STORAGE_SERVICE_KEY**: API key for API authentication with the storage service. Example: dfe83300db5f05f63157f772820bb028bd4d0e27 - * **REPORT_URL**: +* **REPORT_URL**: The base URL to the remote service to which scan reports will be POSTed. - * **REPORT_USERNAME**: +* **REPORT_USERNAME**: Username for API authentication with the reporting service. Not all reporting services require API authentication; leave this unset if API access is unauthenticated. - * **REPORT_PASSWORD**: +* **REPORT_PASSWORD**: Password for API authentication with the reporting service; see above.