[major] IceProd 3.0

[dsschult]

This release removes support for IceProd pilots, and instead relies on HTCondor file transfer and running the module code directly in the HTCondor job. As a consequence, we are able to fully support running in containers via HTCondor.

Breaking changes

• Data transfer now happens at the task level. Other file transfer is deprecated, and will be merged with the task.
• Dataset config removals:
  • Module running_class support has been removed. Use the src attribute instead.
  • Data compression support has been removed.
  • Resources have been removed - use data in permanent input mode.
  • Steering/system has been removed.
• The debugging pilot has limited support, and all data transfer has been removed.
• The dataset truncated status has been moved to an attribute.
• Tasks start at idle instead of waiting.
• Task status waiting is now "ready to queue" and status queued is "on the HTCondor queue" with processing actually being processing in HTCondor.
• The task reset status has been removed, and tasks will now directly go to idle.

Core Checklist

• config as json with validation
• render task as shell script of modules
• get list of input/output files for a task
• file execute parameter = maybe => transfer=???

Server Checklist

• move queued and processing task actions "up" one level to fix queued and processing states
• remove truncated dataset state in favor of a separate flag
• add proper state machine for tasks, jobs, datasets
• update website with new states
• update rest API with new states
• update scheduled tasks with new states
• modify condor submit to use new task shell script as the executable, and condor file transfer as much as possible
• make condor file transfer plugin for gridftp/http?
• update condor plugin to get task resources from condor log

Tests

• new core
• REST routes
• website handlers
• server / batch job submission