Skip to content

Commit

Permalink
Merge pull request #553 from MITLibraries/maintenance-week
Browse files Browse the repository at this point in the history
Maintenance week
  • Loading branch information
ehanson8 authored Jun 6, 2024
2 parents c10fd11 + 0fdc58b commit 4915456
Show file tree
Hide file tree
Showing 21 changed files with 5,441 additions and 6,436 deletions.
2 changes: 1 addition & 1 deletion .python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.11.0
3.12
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.11-slim AS py3
FROM python:3.12-slim AS py3
RUN pip install pipenv

FROM py3 AS wheel
Expand Down
8 changes: 4 additions & 4 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,21 @@ click = "*"
sentry-sdk = "*"
sickle = "*"
smart-open = {extras = ["s3"], version = "*"}
urllib3 = "==1.26.18"
urllib3 = "*"

[dev-packages]
bandit = "*"
black = "*"
coveralls = "*"
mypy = "*"
pre-commit = "*"
pytest = "*"
ruff = "*"
safety = "*"
types-requests = "*"
vcrpy = "==4.2.1"
vcrpy = "*"

[requires]
python_version = "3.11"
python_version = "3.12"

[scripts]
oai = "python -c \"from harvester.cli import main; main()\""
1,349 changes: 932 additions & 417 deletions Pipfile.lock

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion harvester/utils.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
"""utils.py module."""

from typing import Literal

import sentry_sdk


def send_sentry_message(
message: str,
scopes: dict | None = None,
level: str = "warning",
level: Literal["fatal", "critical", "error", "warning", "info", "debug"] = "warning",
) -> str | None:
"""Send message directly to Sentry.
Expand Down
26 changes: 14 additions & 12 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,15 @@ ignore_missing_imports=true
log_level = "INFO"

[tool.ruff]
target-version = "py311"
target-version = "py312"

# set max line length
line-length = 90

# enumerate all fixed violations
show-fixes = true

[tool.ruff.lint]
select = ["ALL", "PT"]

ignore = [
Expand Down Expand Up @@ -47,27 +55,21 @@ ignore = [
# allow autofix behavior for specified rules
fixable = ["E", "F", "I", "Q"]

# set max line length
line-length = 90

# enumerate all fixed violations
show-fixes = true

[tool.ruff.flake8-annotations]
[tool.ruff.lint.flake8-annotations]
mypy-init-return = true

[tool.ruff.flake8-pytest-style]
[tool.ruff.lint.flake8-pytest-style]
fixture-parentheses = false

[tool.ruff.per-file-ignores]
[tool.ruff.lint.per-file-ignores]
"tests/**/*" = [
"ANN",
"ARG001",
"S101",
]

[tool.ruff.pycodestyle]
[tool.ruff.lint.pycodestyle]
max-doc-length = 90

[tool.ruff.pydocstyle]
[tool.ruff.lint.pydocstyle]
convention = "google"
792 changes: 430 additions & 362 deletions tests/fixtures/vcr_cassettes/cli-all-options-except-set-spec.yaml

Large diffs are not rendered by default.

19 changes: 9 additions & 10 deletions tests/fixtures/vcr_cassettes/get-identifiers-no-matches.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,18 @@ interactions:
Connection:
- keep-alive
User-Agent:
- python-requests/2.27.1
- python-requests/2.32.3
method: GET
uri: https://dspace.mit.edu/oai/request?metadataPrefix=oai_dc&from=2021-12-26&until=2021-12-26&set=hdl_1721.1_49432&verb=ListIdentifiers
response:
body:
string: !!binary |
H4sIAAAAAAAAA4ySQWvcMBCF/4rQ3ZasbNLW2A6FUhrYbUO7hZLLokqzscCWXM1s1vvvO3aTQw6F
gA6D9D29eSM1t/M4iCfIGFJsZVVqKSC65EN8bOXP/efivbztmoUqkC4DYA9Agi4TtJJgJjXjIEWf
4dhKJEvBqZUrl31Wfvt4V9zvvgi+IGIre6KpVup8Ppdpgmiz68MTYJnyo2JUmVIr+Q+uZwyvBOer
FTNaV+rXbvvD9TDaIkS2jQ5YhaHGdXObHHeyBHqLn3gL9JyDY3nZNRlwShHhkyXojDam0Jui2uyr
61qb+vrmoVGvEBb8OQHSMunfrdwGpDsPkcIx8OilGIGst2TveY5hbmWy4eCdFMecxlayQVVUpjA3
e63rdT1IcWL58L9DBOLwfjhU70xVVofNh82Vkd2SFDmqx8k6KMdAJfiTUuynnltsXoqugZxTFvwZ
+K1j+g4uZY87S66X3dckxqUCFEeGqAfBqnxp1KrqmpeJdX8BAAD//wMAUjJUR2MCAAA=
string: <?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl"
href="static/style.xsl"?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/
http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2024-06-03T19:51:11Z</responseDate><request
verb="ListIdentifiers" metadataPrefix="oai_dc" from="2021-12-26T00:00:00Z"
until="2021-12-26T00:00:00Z" set="hdl_1721.1_49432">https://dspace.mit.edu//oai/request</request><error
code="noRecordsMatch">No matches for the query</error></OAI-PMH>
headers:
Connection:
- close
Expand All @@ -30,7 +29,7 @@ interactions:
Content-Type:
- text/xml;charset=UTF-8
Date:
- Thu, 14 Apr 2022 15:02:56 GMT
- Mon, 03 Jun 2024 19:51:11 GMT
Set-cookie:
- HttpOnly;Secure
Strict-Transport-Security:
Expand Down
139 changes: 17 additions & 122 deletions tests/fixtures/vcr_cassettes/get-identifiers.yaml

Large diffs are not rendered by default.

70 changes: 9 additions & 61 deletions tests/fixtures/vcr_cassettes/get-records-exclude-deleted.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,18 @@ interactions:
Connection:
- keep-alive
User-Agent:
- python-requests/2.27.1
- python-requests/2.32.3
method: GET
uri: https://dspace.mit.edu/oai/request?metadataPrefix=oai_dc&from=2017-12-14&until=2017-12-14&verb=ListIdentifiers
response:
body:
string: !!binary |
H4sIAAAAAAAAA4ySX2vbMBTFv4rQu60/SevWyCqDMVZIWaEpjL4ETbqpBbblWUrjfvtdJ03WdKwt
6EFIv3vuPUdSV2PbkCcYog9dRUXOKYHOBue7x4reL79lF/RKq4nKYnpuINYAiaTnHiqaYExsjA0l
9QDrisZkkrdsx+XTOVb++HKd3d58JyjQxYrWKfUlY9vtNg89dGawtX+CmIfhkSHKZM4Z3cPlGP1J
wXa2wyTngv28WdzZGlqT+Q7bdhawKvoy7g4XweIkk6HP9COfgV58oC1HtRog9qGL8NUk0JJLmfF5
Ji6X4qzkRSnPHxQ7QbDg9wZimpL+VdGFj+naQZf82mP0lLSQjDPJ3GKOfqxoMH7lLCXrIbQVlVwU
mZCZmC85L3frgZINljf/udSTpYieXOyNhbz1KQe3YQyF2css6rDR6s08WtVgHAxketANvpqDBhJM
xv2R0ihVnsqXopAiF0wIWczPFXsFK3SHrUzb61cDT3HNyrNLjOvvvYqQ7nqw2oZ2tVdcFRd8ptjh
4kjUrvmAsOFIzKeP877IW4Ttc8DNPwkdPoT+AwAA//8DAGpVOzRCAwAA
string: <?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl"
href="static/style.xsl"?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/
http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2024-06-03T19:51:11Z</responseDate><request
verb="ListIdentifiers" metadataPrefix="oai_dc" from="2017-12-14T00:00:00Z"
until="2017-12-14T00:00:00Z">https://dspace.mit.edu//oai/request</request><ListIdentifiers><header
status="deleted"><identifier>oai:dspace.mit.edu:1721.1/112746</identifier><datestamp>2017-12-14T15:03:59Z</datestamp><setSpec>com_1721.1_7803</setSpec><setSpec>hdl_1721.1_7803</setSpec><setSpec>col_1721.1_42001</setSpec><setSpec>hdl_1721.1_42001</setSpec></header></ListIdentifiers></OAI-PMH>
headers:
Connection:
- close
Expand All @@ -31,57 +29,7 @@ interactions:
Content-Type:
- text/xml;charset=UTF-8
Date:
- Tue, 19 Apr 2022 15:07:26 GMT
Set-cookie:
- HttpOnly;Secure
Strict-Transport-Security:
- max-age=63072000
Transfer-Encoding:
- chunked
Vary:
- Accept-Encoding
X-Content-Type-Options:
- nosniff
X-Frame-Options:
- SAMEORIGIN
X-XSS-Protection:
- 1; mode=block
status:
code: 200
message: OK
- request:
body: null
headers:
Accept:
- '*/*'
Accept-Encoding:
- gzip, deflate
Connection:
- keep-alive
User-Agent:
- python-requests/2.27.1
method: GET
uri: https://dspace.mit.edu/oai/request?identifier=oai%3Adspace.mit.edu%3A1721.1%2F112746&metadataPrefix=oai_dc&verb=GetRecord
response:
body:
string: !!binary |
H4sIAAAAAAAAA4yS3U7cMBCFXyXyfeLYuxCIHKNKVX8kUFGhUsXNytgDsZTYaTzLhrevnbAL24vC
VSLPd2bmHFtcTH2XPcEYrHcNYUVJMnDaG+seG/Lr9kt+Ri6kSFQe8LmD0AJghs8DNARhQjqFjmTt
CA8NCajQajpzRTqPyh+fvufXV9+y2MCFhrSIQ03pbrcr/ABOjbq1TxAKPz7SiFJelJQscD0FeyTY
rWaMlyWjv68ub3QLvcqti2OdhqgKtg7z4aXXcZNk6CPzso9ALz6iLUOkGCEM3gX4rBAkLznPy3XO
zm/ZSV1WNT+9E/QIiYI/WwiYkr5vyFfAn6D9aEhmDTi0DxbGhnhlaxMGpaHoLRZgtjWrOCsYZYxX
61OS9YDKKFTXMW87zYqN0UQmByFaOFZTGuv0ZbTY/0hxGJ/2Wr4tKANjlm5wG6/JQAcIyenrfvK9
9QR9A4u4Zhym+iHmw6qc8Zyt53xW9cl5zOe1LgLgzQBaat9vlo6b6qxcCbovHIjWdO8Q2h+IdXop
/2/yL0KXHGQKawmGvglr/wjkXwAAAP//AwDL5/MDNgMAAA==
headers:
Connection:
- close
Content-Encoding:
- gzip
Content-Type:
- text/xml;charset=UTF-8
Date:
- Tue, 19 Apr 2022 15:07:26 GMT
- Mon, 03 Jun 2024 19:51:11 GMT
Set-cookie:
- HttpOnly;Secure
Strict-Transport-Security:
Expand Down
39 changes: 17 additions & 22 deletions tests/fixtures/vcr_cassettes/get-records-include-deleted.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,18 @@ interactions:
Connection:
- keep-alive
User-Agent:
- python-requests/2.27.1
- python-requests/2.32.3
method: GET
uri: https://dspace.mit.edu/oai/request?metadataPrefix=oai_dc&from=2017-12-14&until=2017-12-14&verb=ListIdentifiers
response:
body:
string: !!binary |
H4sIAAAAAAAAA4ySX2vbMBTFv4rQu60/TpZWyCqDMVZIWaEZjL4ETbqpBbbkWUrjfvvJSZM1He0K
ehDS7557z5Hk1di16BGG6IKvMSspRuBNsM4/1PjH6mtxga+UnKgipqcWYgOQUHrqocYJxkTG2GLU
DLCpcUw6OUP2XDmd58rvn6+L25tvKAv4WOMmpV4QstvtytCD14Np3CPEMgwPJKOEl5TgAyzG6M4K
dtUe45Qy8vNmeWca6HThfG7rDeSq6ETcHy6DyZNMhj7SD30EevaRbVms5ACxDz7CF51Accp5QWcF
u1wxLuaVqOi9JGdILvi9hZimpH/VeOliurbgk9u4HD1GHSRtddK3OUc31jhot7YGo80QuhpzyhYF
4wWbrSgV+3WP0TaXt29cqslSzJ5s7LWBsnOpBLslJAuT51nkcaPkq3mUbEBbGND0oNv8ahZaSDAZ
dydKZSlxLi/YgrOSEcb4YvZJkhewzO5yK9316sXAbC5oJeaXOa6/9zJCuuvBKBO69UFxvbiglSTH
ixPR2PY/hAknYjZ9nPdFXiPkkEPe/JPQ8UOoPwAAAP//AwAGRRYMQgMAAA==
string: <?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl"
href="static/style.xsl"?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/
http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2024-06-03T19:51:11Z</responseDate><request
verb="ListIdentifiers" metadataPrefix="oai_dc" from="2017-12-14T00:00:00Z"
until="2017-12-14T00:00:00Z">https://dspace.mit.edu//oai/request</request><ListIdentifiers><header
status="deleted"><identifier>oai:dspace.mit.edu:1721.1/112746</identifier><datestamp>2017-12-14T15:03:59Z</datestamp><setSpec>com_1721.1_7803</setSpec><setSpec>hdl_1721.1_7803</setSpec><setSpec>col_1721.1_42001</setSpec><setSpec>hdl_1721.1_42001</setSpec></header></ListIdentifiers></OAI-PMH>
headers:
Connection:
- close
Expand All @@ -31,7 +29,7 @@ interactions:
Content-Type:
- text/xml;charset=UTF-8
Date:
- Tue, 19 Apr 2022 12:53:30 GMT
- Mon, 03 Jun 2024 19:51:11 GMT
Set-cookie:
- HttpOnly;Secure
Strict-Transport-Security:
Expand Down Expand Up @@ -59,20 +57,17 @@ interactions:
Connection:
- keep-alive
User-Agent:
- python-requests/2.27.1
- python-requests/2.32.3
method: GET
uri: https://dspace.mit.edu/oai/request?identifier=oai%3Adspace.mit.edu%3A1721.1%2F112746&metadataPrefix=oai_dc&verb=GetRecord
response:
body:
string: !!binary |
H4sIAAAAAAAAA4ySX2/bIBTFvwri3cbgZGkRppo07Y/UatWaSlNfIga3NZINniGN++0HcZM2e2j7
ZIv7O/fec0BcTH2HHmEM1rsG07LCCJz2xrqHBt+uvxZn+EKKTBUhPnUQWoCI4tMADY4wRTKFDqN2
hPsGh6ii1WTPlfk8KX9+/lFcX31HqYELDW5jHDghu92u9AM4NerWPkIo/fhAEkpYWRE8w3wK9kSw
q/cYqypKfl9d3ugWelVYl8Y6DUkVLA/7w0uv0ybZ0EfmoY9Azz6SLYOlGCEM3gX4oiJIVjFWVIuC
nq8p48ua1/ROkBMkCf5uIcSc9J8Gf4P4C7QfDUbWgIv23sLYYK8sN2FQGsrexhLMltMVoyUllLLV
4hNGPURlVFTXKW877RUbo7HMDkKycKomJNXJ82hx+JHiOD7vNX9bUAZGlG9wm67JQAcRstOX/eR7
6wnyChZpzTRM9UPKh64Kygq6WNMlr2q+PE/5vNRFgHgzgJba95u542Z1VtWCHApHojXdO4T2R2KR
X8rbTf5HyJyDzGHNwZBXYR0egfwHAAD//wMARuZMJDYDAAA=
string: <?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl"
href="static/style.xsl"?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/
http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2024-06-03T19:51:11Z</responseDate><request
verb="GetRecord" identifier="oai:dspace.mit.edu:1721.1/112746" metadataPrefix="oai_dc">https://dspace.mit.edu//oai/request</request><GetRecord><record><header
status="deleted"><identifier>oai:dspace.mit.edu:1721.1/112746</identifier><datestamp>2017-12-14T15:03:59Z</datestamp><setSpec>com_1721.1_7803</setSpec><setSpec>hdl_1721.1_7803</setSpec><setSpec>col_1721.1_42001</setSpec><setSpec>hdl_1721.1_42001</setSpec></header></record></GetRecord></OAI-PMH>
headers:
Connection:
- close
Expand All @@ -81,7 +76,7 @@ interactions:
Content-Type:
- text/xml;charset=UTF-8
Date:
- Tue, 19 Apr 2022 12:53:31 GMT
- Mon, 03 Jun 2024 19:51:11 GMT
Set-cookie:
- HttpOnly;Secure
Strict-Transport-Security:
Expand Down
Loading

0 comments on commit 4915456

Please sign in to comment.