Code

[major] IceProd 3.0#359

Code

Merged

dsschult merged 162 commits intomasterWIPACrepo/iceprod:masterfrom singularityWIPACrepo/iceprod:singularityCopy head branch name to clipboardOct 1, 2024

+11,720-12,321

Commits

Commits on Apr 19, 2024

• dataclass changes

  dsschult

  committed
  78ca9b8

  Copy full SHA for 78ca9b8

• new json schema for configs, with tests

  dsschult

  committed
  602958e

  Copy full SHA for 602958e

• remove get_pkgdata_filename in favor of importlib.resources.files

  dsschult

  committed
  4c9113f

  Copy full SHA for 4c9113f

• more config and tests

  dsschult

  committed
  1c76618

  Copy full SHA for 1c76618

• write a config to file, with tests

  dsschult

  committed
  2de67b0

  Copy full SHA for 2de67b0

• new i3exec

  dsschult

  committed
  0856c1a

  Copy full SHA for 0856c1a

• <bot> update setup.cfg

  

  github-actions

  authored and

  dsschult

  committed
  6bd9eb8

  Copy full SHA for 6bd9eb8

• <bot> update requirements-docs.txt

  

  github-actions

  authored and

  dsschult

  committed
  3ac88c9

  Copy full SHA for 3ac88c9

• <bot> update requirements-tests.txt

  

  github-actions

  authored and

  dsschult

  committed
  3074b75

  Copy full SHA for 3074b75

• <bot> update requirements.txt

  

  github-actions

  authored and

  dsschult

  committed
  cc89a86

  Copy full SHA for cc89a86

• clean up core tests

  dsschult

  committed
  3edb62b

  Copy full SHA for 3edb62b

• new dataset/job/task states, and updated rest routes

  dsschult

  committed
  2c92c9b

  Copy full SHA for 2c92c9b

• <bot> update requirements-docs.txt

  

  github-actions

  authored and

  dsschult

  committed
  38f9d25

  Copy full SHA for 38f9d25

• <bot> update requirements-tests.txt

  

  github-actions

  authored and

  dsschult

  committed
  e65ff70

  Copy full SHA for e65ff70

• <bot> update requirements.txt

  

  github-actions

  authored and

  dsschult

  committed
  1aa2738

  Copy full SHA for 1aa2738

• move scheduled_tasks out of the server submodule, since it no longer depends on it

  dsschult

  committed
  388ab41

  Copy full SHA for 388ab41

• website redo, with modified REST action handling

  dsschult

  committed
  b4722c0

  Copy full SHA for b4722c0

• new server config

  dsschult

  committed
  4c77386

  Copy full SHA for 4c77386

• updates with new tests. not finished

  dsschult

  committed
  9fe0d55

  Copy full SHA for 9fe0d55

• all new server code and tests

  dsschult

  committed
  1ea105e

  Copy full SHA for 1ea105e

• <bot> update setup.cfg

  

  github-actions

  authored and

  dsschult

  committed
  50f9816

  Copy full SHA for 50f9816

• <bot> update requirements-docs.txt

  

  github-actions

  authored and

  dsschult

  committed
  6f94704

  Copy full SHA for 6f94704

• <bot> update requirements-tests.txt

  

  github-actions

  authored and

  dsschult

  committed
  bf91b71

  Copy full SHA for bf91b71

• <bot> update requirements.txt

  

  github-actions

  authored and

  dsschult

  committed
  2d941a4

  Copy full SHA for 2d941a4

• get all tests passing

  dsschult

  committed
  b952e24

  Copy full SHA for b952e24

• <bot> update requirements-docs.txt

  

  github-actions

  authored and

  dsschult

  committed
  f26354c

  Copy full SHA for f26354c

• <bot> update requirements-tests.txt

  

  github-actions

  authored and

  dsschult

  committed
  88e6ff7

  Copy full SHA for 88e6ff7

• <bot> update requirements.txt

  

  github-actions

  authored and

  dsschult

  committed
  5ebce36

  Copy full SHA for 5ebce36

• <bot> update setup.cfg

  

  github-actions

  committed
  92d13df

  Copy full SHA for 92d13df

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  06c1a78

  Copy full SHA for 06c1a78

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  b94976e

  Copy full SHA for b94976e

• <bot> update requirements.txt

  

  github-actions

  committed
  f485b9d

  Copy full SHA for f485b9d

• fix requirements to include condor

  dsschult

  committed
  23471f5

  Copy full SHA for 23471f5

• <bot> update setup.cfg

  

  github-actions

  committed
  06ee32d

  Copy full SHA for 06ee32d

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  512be0e

  Copy full SHA for 512be0e

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  4d06e0e

  Copy full SHA for 4d06e0e

• <bot> update requirements.txt

  

  github-actions

  committed
  ff2fbd0

  Copy full SHA for ff2fbd0

• bump python version in CI

  dsschult

  committed
  88c1294

  Copy full SHA for 88c1294

• fix Dockerfile

  dsschult

  committed
  c6f2dc1

  Copy full SHA for c6f2dc1

• <bot> update setup.cfg

  

  github-actions

  committed
  217aeea

  Copy full SHA for 217aeea

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  d5c12d2

  Copy full SHA for d5c12d2

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  ef7b108

  Copy full SHA for ef7b108

• <bot> update requirements.txt

  

  github-actions

  committed
  7318993

  Copy full SHA for 7318993

Commits on May 30, 2024

• parallelize temp cleaning to go faster

  dsschult

  committed
  8a34222

  Copy full SHA for 8a34222

• handle when task stats are 0

  dsschult

  committed
  5b1da7f

  Copy full SHA for 5b1da7f

• allow specifying the REST API url via an env variable

  dsschult

  committed
  25f2be3

  Copy full SHA for 25f2be3

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  380548d

  Copy full SHA for 380548d

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  b51774c

  Copy full SHA for b51774c

• <bot> update requirements.txt

  

  github-actions

  committed
  7c7a4d1

  Copy full SHA for 7c7a4d1

Commits on Jun 3, 2024

• fix job_temp_cleaning script to await futures

  dsschult

  committed
  335fc1c

  Copy full SHA for 335fc1c

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  2077b80

  Copy full SHA for 2077b80

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  f5fe501

  Copy full SHA for f5fe501

• <bot> update requirements.txt

  

  github-actions

  committed
  374caa4

  Copy full SHA for 374caa4

• update credentials to use new rest-tools syntax

  dsschult

  committed
  52f991f

  Copy full SHA for 52f991f

• try setting the timezone in github CI, to see if that fixes the HTCondor tests

  dsschult

  committed
  832e758

  Copy full SHA for 832e758

Commits on Jun 5, 2024

• convert to UTC for HTCondor log files, and update the tests

  dsschult

  committed
  1358822

  Copy full SHA for 1358822

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  02b5244

  Copy full SHA for 02b5244

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  12715d7

  Copy full SHA for 12715d7

• <bot> update requirements.txt

  

  github-actions

  committed
  6750a16

  Copy full SHA for 6750a16

• build docs with python 3.12

  dsschult

  committed
  d91dc52

  Copy full SHA for d91dc52

Commits on Jun 10, 2024

• website updates

  dsschult

  committed
  a564fa5

  Copy full SHA for a564fa5

• build container to cvmfs

  dsschult

  committed
  0ad628e

  Copy full SHA for 0ad628e

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  3b438ad

  Copy full SHA for 3b438ad

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  f16ca94

  Copy full SHA for f16ca94

• <bot> update requirements.txt

  

  github-actions

  committed
  35ee3ff

  Copy full SHA for 35ee3ff

• fix web test

  dsschult

  committed
  0fb5737

  Copy full SHA for 0fb5737

• forgot runs-on

  dsschult

  committed
  6efe873

  Copy full SHA for 6efe873

• allow empty task_info

  dsschult

  committed
  4484140

  Copy full SHA for 4484140

• set rest clients in queue

  dsschult

  committed
  e2f5ac3

  Copy full SHA for e2f5ac3

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  77003b3

  Copy full SHA for 77003b3

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  89ec514

  Copy full SHA for 89ec514

• <bot> update requirements.txt

  

  github-actions

  committed
  402a3fe

  Copy full SHA for 402a3fe

Commits on Jun 11, 2024

• fix task queueing to move from idle to waiting

  dsschult

  committed
  9652a73

  Copy full SHA for 9652a73

• fix flake8

  dsschult

  committed
  0a996a9

  Copy full SHA for 0a996a9

• write sha into image name for cvmfs

  dsschult

  committed
  b94d704

  Copy full SHA for b94d704

• write sha into image name for cvmfs v2

  dsschult

  committed
  25df73e

  Copy full SHA for 25df73e

Commits on Jun 12, 2024

• POST is the right method

  dsschult

  committed
  6a37c3c

  Copy full SHA for 6a37c3c

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  1d1c90d

  Copy full SHA for 1d1c90d

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  0d7684d

  Copy full SHA for 0d7684d

• <bot> update requirements.txt

  

  github-actions

  committed
  4c72004

  Copy full SHA for 4c72004

• fix task javascript

  dsschult

  committed
  3de7528

  Copy full SHA for 3de7528

• need to set instance id

  dsschult

  committed
  c93aff4

  Copy full SHA for c93aff4

Commits on Aug 20, 2024

• change condor submit interface to be more future-compatible

  dsschult

  committed
  c2a2c0d

  Copy full SHA for c2a2c0d

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  5e7a015

  Copy full SHA for 5e7a015

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  0b1fc37

  Copy full SHA for 0b1fc37

• <bot> update requirements.txt

  

  github-actions

  committed
  8c37ff8

  Copy full SHA for 8c37ff8

• add some debugging

  dsschult

  committed
  cbdecdb

  Copy full SHA for cbdecdb

Commits on Aug 21, 2024

• try again to get submit classads correct

  dsschult

  committed
  7f2cc69

  Copy full SHA for 7f2cc69

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  57ae805

  Copy full SHA for 57ae805

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  9656f16

  Copy full SHA for 9656f16

• <bot> update requirements.txt

  

  github-actions

  committed
  cacc24e

  Copy full SHA for cacc24e

• fix infiles and outfiles

  dsschult

  committed
  cdcce74

  Copy full SHA for cdcce74

• logging for transfer plugins

  dsschult

  committed
  076d7a3

  Copy full SHA for 076d7a3

Commits on Aug 22, 2024

• fix legacy submit to use itemdata

  dsschult

  committed
  87b4a23

  Copy full SHA for 87b4a23

• POST to complete tasks

  dsschult

  committed
  a773de7

  Copy full SHA for a773de7

• when restarting, reload jobs into memory

  dsschult

  committed
  9fcffce

  Copy full SHA for 9fcffce

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  9264ec7

  Copy full SHA for 9264ec7

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  4ad1124

  Copy full SHA for 4ad1124

• <bot> update requirements.txt

  

  github-actions

  committed
  f279269

  Copy full SHA for f279269

• upload stdout and stderr files

  dsschult

  committed
  19ec2b4

  Copy full SHA for 19ec2b4

• fix log name

  dsschult

  committed
  cba74a6

  Copy full SHA for cba74a6

Commits on Aug 23, 2024

• retry gridftp transfers

  dsschult

  committed
  7007ced

  Copy full SHA for 7007ced

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  f865e4d

  Copy full SHA for f865e4d

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  819e4ba

  Copy full SHA for 819e4ba

• <bot> update requirements.txt

  

  github-actions

  committed
  04c241c

  Copy full SHA for 04c241c

• remove duplicate JELs

  dsschult

  committed
  3a32dda

  Copy full SHA for 3a32dda

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  be5252c

  Copy full SHA for be5252c

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  9570e41

  Copy full SHA for 9570e41

• <bot> update requirements.txt

  

  github-actions

  committed
  4e53dc6

  Copy full SHA for 4e53dc6

Commits on Aug 27, 2024

• make a shorter spin cycle for materialization, so other datasets can materialize things too instead of one hogging it all

  dsschult

  committed
  4675fb1

  Copy full SHA for 4675fb1

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  a3c4f8e

  Copy full SHA for a3c4f8e

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  9d362d4

  Copy full SHA for 9d362d4

• <bot> update requirements.txt

  

  github-actions

  committed
  d8c2da8

  Copy full SHA for d8c2da8

• add support for task_files

  dsschult

  committed
  f3366a9

  Copy full SHA for f3366a9

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  c39bf6d

  Copy full SHA for c39bf6d

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  0b3ea91

  Copy full SHA for 0b3ea91

• <bot> update requirements.txt

  

  github-actions

  committed
  8afdaa3

  Copy full SHA for 8afdaa3

Commits on Sep 6, 2024

• add support for maybe file transfer

  dsschult

  committed
  db94627

  Copy full SHA for db94627

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  ab7e913

  Copy full SHA for ab7e913

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  604df0a

  Copy full SHA for 604df0a

• <bot> update requirements.txt

  

  github-actions

  committed
  9cbb240

  Copy full SHA for 9cbb240

• gitattributes to ignore requirements in merge

  dsschult

  committed
  2d45556

  Copy full SHA for 2d45556

• Merge branch 'master' into singularity

  dsschult

  committed
  a976fda

  Copy full SHA for a976fda

• fix tests and add test for maybe and false transfers

  dsschult

  committed
  0383608

  Copy full SHA for 0383608

• fix condor tests and add test for maybe and false transfers

  dsschult

  committed
  105d0d3

  Copy full SHA for 105d0d3

• maybe should be all caps

  dsschult

  committed
  7da572f

  Copy full SHA for 7da572f

• allow 0 as false

  dsschult

  committed
  33d28b5

  Copy full SHA for 33d28b5

Commits on Sep 10, 2024

• fix truncated status. fix bad 400s for task/job/dataset already in correct status

  dsschult

  committed
  c6b22e9

  Copy full SHA for c6b22e9

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  0770a8e

  Copy full SHA for 0770a8e

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  562d617

  Copy full SHA for 562d617

• <bot> update requirements.txt

  

  github-actions

  committed
  e38607c

  Copy full SHA for e38607c

• fix return codes for latest commit

  dsschult

  committed
  18d7261

  Copy full SHA for 18d7261

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  5bb3803

  Copy full SHA for 5bb3803

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  f3dc2a3

  Copy full SHA for f3dc2a3

• <bot> update requirements.txt

  

  github-actions

  committed
  037e2f2

  Copy full SHA for 037e2f2

• fix keyerror in materialization. add more tests

  dsschult

  committed
  6a07c5a

  Copy full SHA for 6a07c5a

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  4962fa7

  Copy full SHA for 4962fa7

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  d4f4a50

  Copy full SHA for d4f4a50

• <bot> update requirements.txt

  

  github-actions

  committed
  4e92003

  Copy full SHA for 4e92003

• add hard reset action to task web pages

  dsschult

  committed
  69b734e

  Copy full SHA for 69b734e

• fix test

  dsschult

  committed
  1a3d0e0

  Copy full SHA for 1a3d0e0

• fix edge cases when resetting/truncating dataset

  dsschult

  committed
  dfd7da2

  Copy full SHA for dfd7da2

Commits on Sep 12, 2024

• add script to convert from v2 configs

  dsschult

  committed
  4752f48

  Copy full SHA for 4752f48

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  9836ec7

  Copy full SHA for 9836ec7

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  657ff25

  Copy full SHA for 657ff25

• <bot> update requirements.txt

  

  github-actions

  committed
  55971a2

  Copy full SHA for 55971a2

Commits on Sep 20, 2024

• allow running_class to live on for a bit longer, to ease the transition

  dsschult

  committed
  da317be

  Copy full SHA for da317be

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  eab4318

  Copy full SHA for eab4318

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  431b2c1

  Copy full SHA for 431b2c1

• <bot> update requirements.txt

  

  github-actions

  committed
  4edb9c5

  Copy full SHA for 4edb9c5

• fix test

  dsschult

  committed
  c35ea5e

  Copy full SHA for c35ea5e

Commits on Sep 30, 2024

• make sure job_temp is set

  dsschult

  committed
  e63130c

  Copy full SHA for e63130c

• <bot> update requirements-docs.txt

  

  github-actions

  committed
  9da6604

  Copy full SHA for 9da6604

• <bot> update requirements-tests.txt

  

  github-actions

  committed
  1a7c05b

  Copy full SHA for 1a7c05b

• <bot> update requirements.txt

  

  github-actions

  committed
  b8221a3

  Copy full SHA for b8221a3

• expand storage location to get correct url

  dsschult

  committed
  312bc8d

  Copy full SHA for 312bc8d

• remove timezone from string format

  dsschult

  committed
  fd1afa2

  Copy full SHA for fd1afa2

• fix flake8

  dsschult

  committed
  f206a41

  Copy full SHA for f206a41

Commits on Oct 1, 2024

• fix rest handler not returning after error

  dsschult

  committed
  ae3b11e

  Copy full SHA for ae3b11e

• fix flake8

  dsschult

  committed
  10e20dd

  Copy full SHA for 10e20dd

• fix roll file test at day boundary

  dsschult

  committed
  edb4686

  Copy full SHA for edb4686

• fix batchsys iteration

  dsschult

  committed
  d0075b9

  Copy full SHA for d0075b9