Scheduled monthly dependency update for July

[pyup-bot]

Update toil[cwl] from 7.0.0 to 7.0.0.

Changelog

6.1.0

Highlighted Features Added

* WDL and CWL task standard output and standard error logs that are not captured by the workflow will now be logged at INFO level and stored in the `--writeLogs`/`--writeLogsGzip` directory. (4657)
* Use a default log limit of 100MiB (4788)

Breaking Changes

* Stats and logging system again uses job display name (4755)
* `--disableProgress` is once again a flag that doesn't take an argument (4758)


CWL

* Don't clear out user-provided values for the --default-container option (4730)

WDL

* WDL job names now include numbers for scatters (4755)
* Multi-line WDL placeholder substitutions no longer interfere with de-indenting WDL command blocks (https://github.com/chanzuckerberg/miniwdl/issues/665)
* Standard error for failed tasks is now always logged to the worker log somewhere (4781)

Kubernetes

Dependencies

* Deps: removed the ruaml.yaml.string plugin dependency for a simpler solution (4760)

Misc

* Toil will no longer warn about a missing XDG_RUNTIME_DIR (4769)
* Read the Docs and CI docs builds should have Graphviz installed (pending CI image rebuild) (4734)
* Add more Python3.12 compatibility by replacing the one function from distutils that we use, `strtobool()`. (4765)
* Set default cache folders to be accessible between toil-wdl-runner workflows (Same as MiniWDL/Singularity defaults) (4761)
* Set toil-wdl-runner cache folders on Toil managed clusters to be at `/var/lib/toil` (4761)
* Fall back to assuming machine has 1 core when CPU count is unavailable. (4545)
* `FileJobStore` now supports filenames that get modified when percent-encoded (4779)

Thank you to our contributors:

DailyDreaming mr-c stxue1 adamnovak app/dependabot 

**Full Changelog**: https://github.com/DataBiosphere/toil/compare/releases/6.0.0...releases/6.1.0

6.0.0

NOTE!
We now have a config file!  https://toil.readthedocs.io/en/latest/running/cliOptions.html#the-config-file


Breaking Changes
* Removed the parasol batch system
* Removed the TES batch system (this is now a plugin)
* Removed our WDL compiler in favor of an interpreter (we still support WDL, we just do it differently now)
* We no longer support python3.7


CWL
* Support CWL 1.2.1 (4682)
* CWL Pipefish compatibility (4636)
* Support per-task preemptibility in CWL (4551)
* Fix configargparse in CWL (4618)
* cwl: use the latest commit from the proposed CWL v1.2.1 branch (4565)
* Upgrade cwltool to avoid broken galaxy-tool-util release. (4639)
* Implement a better config file system for CWL/WDL options (4666)
* Allow working with remote files in CWL and WDL workflows (4690)
* Make cwl mutually exclusive groups exist only when cwl is not suppressed (4725)
* Log more usefully for CWL workflows (4736)


WDL
* Simplify WDL Toil job graphs (4524)
* More WDL and Slurm documentation (4558)
* Improve WDL documentation (4732)
* Add String to File functionality into toil-wdl-runner (4589)
* Run WDL output through Toil export system to support URIs (4579)
* Allow the WDL output section to reference itself (4592)
* Ensure sibling files in toil-wdl-runner (4610)
* Make WDLOutputJob collect all task outputs (4602)
* Report errors in WDL using MiniWDL's error location printer (4637)
* Remove the WDL compiler. (4679)
* Implement a better config file system for CWL/WDL options (4666)
* Allow working with remote files in CWL and WDL workflows (4690)
* Strip leading whitespace from WDL commands (4720)


Misc
* Add config file support (4569)
* Support Python3.11 and drop Python 3.7 (4646)
* Move TES batch system to a plugin (4650)
* Turn batch system tests back on (4649)
* Separate out integration tests to run on a schedule (4612)
* Avoid concurrent modification in cluster scaler tests (4600)
* Remove old buckets from AWS (4588)
* Tests: only request a single core (4572)
* Reduce the number of assert statements (4590)
* take any nvidia-smi exception as not having gpu (4611)
* More resiliancy (4395)
* Remove useage of the deprecated pkg_resources (4701)
* Make sure cwltool always knows we have an outdir to fix 4698 (4699)
* AWS jobStoreTest: re-use delete_s3_bucket from toil.lib.aws (4700)
* Only count output file usage when using the file store (4692)
* Remove the parasol batch system. (4678)
* Move around reqs and move aws dev libraries to aws (4664)
* Make sure the `--batchLogsDir` exists if it is set (4635)
* Update EC2 instances and EC2 update script. (4745)
* remove extraneous dependency on old 'mock' (4739)
* Point CI at the new public URLs for stuff we host
* Add __init__.py to options folder (4723)


_Bug Fixes_
* Lower redirect log level to fix 4526 (4578)
* Fix mypy from being broken by new boto types (4577)
* Fix CI on local Gitlab runners (4571)
* Banish ghost jobs (4563)
* Stop deleting chained-to jobs which fail as orphaned jobs (4557)
* Fix pickling error when jobstate file doesnt exist and fix threading error when lock file exists then disappears (4575)
* Fix 3867 and try to explain but not crash when bad things happen to our mutex file (4656)
* Fix CI Appliance Builds (4655)
* Tolerate a failed AMI polling attempt (4727)* Add pure Python fallback for getDirSizeRecursively() (4753)
* Don't mark inputs (or outputs) executable for no reason (4728)
* Fix scheduled CI tests (4742)
* Fix --printJobInfo (4709)


Thank you to our contributors: stxue1 , w-gao, DailyDreaming , mr-c , adamnovak , glennhickey, misterbrandonwalker, and a-detiste !

5.12.0

WDL
* Virtualize filenames as in-container paths from point of view of WDL command (4527)
* Add WDL conformance tests to CI (4530)
* Use less memory in the Giraffe WDL test (4541)

Version Upgrades
* Upgrade to cwltool 3.1.20230601100705 (4500)
* Update mock requirement from <5,>=4.0.3 to >=4.0.3,<6 (4366)

Misc
* Anonymous access to Google Storage (4518)
* Reorder config so that default settings are applied first (4528)
* Add a way to forward accelerators to Docker containers (4492)

_Bug Fixes_
* Fix test failures without docker installed (4544)
* Prevent certain tests from being run twice in CI (4529)
* Drop external Docker builder (4523)
* Fix CI lint test (4533)
* Grab AWS group policies on top of user (4505)
* Grab accelerator set off the end of the list instead of by index (4506)
* Fix RtD build (4491)
* Include tests (4499)

Thank you to our contributors: stxue1 , DailyDreaming , mr-c , adamnovak , and tjni !

5.11.0

Breaking Changes
* Imported files will be symlinked by default, unless the user sets `--noLinkImports` or the workflow imports with `symlink=False`. (3949)

WDL
* Toil will now stop if it encounters an error polling a possible import URL for a WDL workflow input file. (4479)
* WDL workflows will be protected against imported files with no basenames. (4477)

Misc
* Toil batch system ID numbers for issued jobs now start at 1. (4482)
* Attempts to import files from URLs when the implementing job store is missing an extra are now better reported. (4479)
* Include tests in the source distribution that gets published to PyPI (4499)

_Bug Fixes_
* Toil should no longer crash when a delete wins a race against a load in `FileJobStore` (4484)
* Prevent local root jobs (such as WDLRootJob) from being run twice. (4482)
* Slurm and other grid batch system jobs will now have more informative names (4472)
* WDL workflows can no longer import `""` as a File. (4477)

Thank you to our contributors: stxue1, DailyDreaming, mr-c, adamnovak

5.10.0

Changelog

Highlighted Features Added
* Add a `--caching` option which explicitly states whether to use caching with a workflow. Uses a default value depending on whether or not we are using the file job store if not specified. (4218)
* New prototype WDL runner `python -m toil.wdl.wdltoil` using MiniWDL (3468)
* MiniWDL-based WDL implementation can now run the vg Giraffe WDL workflow ( 4353)
* Toil now tests against our own tiny set of WDL conformance tests (4351)
* Toil can run the HPRC assembly WDL workflows (4435)
* Toil can now use Mesos roles (4455)

Breaking Changes

* Replace "preemptable" with "preemptible", add example of using --defaultPreemptible flag to Preemptibility documentation (1951)

CWL
* CWL: run all ExpressionTools on the Leader node, instead of submitting separate jobs (4157)

Kubernetes
* Kubernetes batch system: Delete jobs individually when batch delete fails (3403)
* Documentation for running a Toil leader for a Kubernetes workflow outside Kubernetes now covers examples and common problems for running CWL workflows (document toil-cwl-runner + "Running the Leader Outside Kubernetes" 3422)
* Kubernetes batch system: support `--maxCores`, `--maxDisk`, and `--maxMemory` (2864)
* Add tutorial for Kubernetes launch cluster (3743)


Dependencies

* Require htcondor 10 exactly (4315)
* Toil jobs now have a `local` parameter which determines if they should run on the leader. (4388)

Misc

* The offline tests can now be run in parallel (3493)
* Code updated to be more idiomatic for Python3.7 (4295)
* Support for a `--network` for `toil launch-cluster` for Google cloud (4196)
* Support for a `--use_private_ip` for `toil launch-cluster` to dial nodes by private IP instead of public IP (4196)
* GPU scheduling should now be supported on Slurm (4308)
* Toil now supports a `--batchLogsDir` option and `TOIL_BATCH_LOGS_DIR` environment variable, to provide a directory other than the work dir where Toil will instruct HPC batch systems to save their captured job logs.
* `htcondor` batch system should now work again, and will retry connections
*  Updated the --coalesceStatusCalls help documentation to reflect the current state of https://github.com/DataBiosphere/toil/issues/4431 (#4437)
* Toil no longer trusts XDG_RUNTIME_DIR under Slurm (fixes some of the issues behind 4395 when Slurm is configured not to follow the XDG spec) (4435)
* Toil now puts it lock files for Singularity cache directories for WDL in those directories (4435)
* Toil's WDL interpreter can now use local-to-the-leader jobs for evaluating WDL code that doesn't need appreciable resources (4388)
* Toil now tolerates more possible exceptions related to the panasas network file system (4440)
* Type hinting to functions in resource.py (938)
* Added return type to inVirtualEnv() in `__init__.py` (938)
* Added None checks to some function bodies (938)



_Bug Fixes_
* Stop crashing when predefined batch job exit reasons are used and need to go into the message bus log file (4321)
* Added `import subprocess` to restore the behavior of 588. (4429)
* Toil will no longer use the stored message bus path from an old execution of a workflow when deciding where to save the message bus log when restarting a workflow (4438)
* Fix --custom-net mutual exclusivity bug. (4458)


Thank you to our contributors: stxue1 , DailyDreaming , mr-c , adamnovak , jfennick , misterbrandonwalker , w-gao , stephanaime , glennhickey , Hexotical , manabuishii gmloose , boukn , and thiagogenez !

5.9.2

Changelog

_Bug Fixes_
* Change build tag import (4329)

Thank you to our contributors: adamnovak , Hexotical !

5.9.0

Changelog

_Bug Fixes_
* Fix --provisioner and --metrics together (4328)
* Ignore incorrect type hint from boto3, remove json.loads (4330)
* Warn about missing --bypass-file-store with in-place update (4337)
* Replace prepareHTSubmission with prepareSubmission in HTCondor (4319)
* Merge "Google fixes" (4293)
* Support (only) current htcondor (4320)
* Delete k8s jobs individually when batch delete fails (4306)

Misc
* Update aws spot documentation (4310)
* Enable parallel testing (3493)
* Add documentation for running CWL workflows on non-Toil-managed Kubernetes clusters (4332)
* Export all slurm args by default (4237)
* Allow for subclasses of base types in messages (4322)
* Non cache default (4299)

Dependencies
* Bump mypy from 0.982 to 0.991 (4345)
* Bump schema-salad>=8.4.20230128170514,<9 to schema-salad>=8.3.20220913105718,<8.4 (4342) (4341)
* Bump cwltool from 3.1.20221008225030 to 3.1.20221201130942 (4338)
* Bump pyupgrade to 3.7 (4295)

Thank you to our contributors: adamnovak , Hexotical , w-gao, mr-c , gmloose , boukn , and thiagogenez !

5.8.0

Changelog

Highlighted Features Added

* Toil server now exposes workflow tasks via WES (4046).
* Toil server now has a `--wes_dialect agc` option that will hide any tasks that don't have Amazon Batch job IDs, and put the IDs in the task names for those that do (4047).
* Toil jobs now accept an `accelerators` requirement, like `accelerators=1` or `accelerators={'kind': 'gpu', 'brand': 'nvidia', 'count': 2}` (4163)
* Include total requested cores for each job type in `toil stats` (4173)
* Toil jobs now expose `job.accelerators` to workflow 
* Add prefix suffix params to `AbstractFileStore.getLocalTempFile` and `AbstractFileStore.getLocalTempFileName` (4273)
* CWL: `--no-compute-checksum`, `--strict-cpu-limit`, `--disable-validate`, and `--fast-parser` are now available


Breaking Changes
* Toil's built-in autoscaler now guesses that some memory and disk space on nodes will not actually be available for jobs; pass `--assumeZeroOverhead` to revert to the old behavior (2103) 

CWL
* CWL job unit and display names have been changed to make more sense as task names, and management of them has been unified into a `CWLNamedJob`. (4046/4047)
* CWL `CUDARequirement` is parsed by `cwltool` and turned into a requirement for the minimum requested number of nvidia GPU accelerators (3982)
* fix false warning when outputSource contains only one None value (4300)


Kubernetes
* `KubernetesBatchSystem` can add `nvidia.com/gpu` and `amd.com/gpu` resource requests for jobs that request those accelerators (4163)
* `KubernetesBatchSystem` can request GPUs by `model` key, if nodes are labeled appropriately (4163)

Dependencies

Misc
* Toil WES server now accepts requests that leave out workflow_params. (4037)
* The `MessageBus` has been expanded to use `pypubsub`, and now has `MessageInbox` and `MessageOutbox` objects to represent connections to it. (4046/4047)
* `ToilMetrics` now rides on the `MessageBus` rails. (4046/4047)
* Toil workflows now have a `--writeMessages` option, which takes a file to which a line-oriented stream of `MessageBus` messages will be written. Reading this file will allow you to recover the current state of the workflow. (4046/4047)
* Add code for warning check to be used when launching cluster with AWS. (3514)
* Use a CI prebake image for gitlab testing. (4185)
* Toil clusters now have `/var/tmp` as the default temporary directory, since they often make large temporary files (4148)
* Adds basic testing for slurm using a slurm docker cluster by running sample workflows. (3856)
* Add message bus documentation (4239)
* `SingleMachineBatchSystem` can schedule nvidia GPU accelerators, limiting the concurrent jobs to no more than there are accelerators to support, and setting `CUDA_VISIBLE_DEVICES` in the tasks' environments to tell them which nvidia GPU(s) to use. (4163)
* `AWSBatchBatchSystem` can use AWS Batch's GPU resource to provide nvidia GPU accelerators (4163)
* Toil jobs no longer need to re-run after their child/followOn/service jobs in order to delete themselves. (3188)
* Message bus is now thread safe (4276)
* Docker build has been updated with new Aventer Mesos deb URL (fixes 4290)
* `docker` binary in the container has been updated to that included in the Ubuntu repos (fixes 4282)
* Singularity in the appliance has been updated to 3.10 which is >=3.9, for cgroups v2 support.
* Base Ubuntu container image for the appliance has been updated to 22.04, which has a new enough libc for Debian's Singularity 3.10 debs.
* Safer type usage checking for systems without boto3 installed
* Tests are now more runnable post-installation. Temporary paths are not selected based upon the location of the tests themselves. (4287)

_Bug Fixes_
* Only use `/var/run/user` if XDG tells us we have it in our session. Otherwise we will try other places, including `/run/lock/toil`. (4170)
* `toil destroy-cluster`: terminate stopped instances when destroying the cluster (4271)
* fileJobStore: handle arbitrary `os.link` errors to work on some filesystems (2232)

Thank you to our contributors!

5.7.1

Changelog


Highlighted Features Added

AWS Batch Batch System (3956)
AGC Integration (4039) + More AGC integration (4067) + AGC megabranch (4113)
Scale TES to be able to run reasonably-sized workflows on Funnel on Kubernetes with the AWS job store (3927)

CWL

Run CWL conformance tests via WES (4052)
Implement and test CWL loadContents from URLs to fix 4125 (4126)
Add CWL tests under ARM (4038)
Cache results of cwltool version lookup (4141)

Misc

SGE batch system change to support serial jobs. (4022)
Performance testing for Graviton instances (4123)
Stop waiting on hostpath volumes to exist (4146)
Catch and warn about jobs going away too slowly on FileJobStore (4149)
Add documentation for the type-checking hooks (4117)
Pod murder bot (4060)
Contrib hook scripts (4105)
Allow newer google-cloud-storage (4114)
Use environment variable to set parallel partition name (4096)
Register pytest markers (4103)
Mention --export=ALL for SLURM environments (4100) (4102)
Allow persisting workflow state in WES server across container recreation (4082)
Change `toil kill` to use the job store shared file API to find `pig.log` (4075)
Bring back kill loop in the single_machine batch system but with a timeout (4070)
Reorganize Locking (4059)
Add and test preemptability constraints (4044)
Enhanced types (3975)
Use an `init` process that reaps zombies on toil clusters (3974)
Add launch cluster support for ARM (3971)
Feat: square bracket to period separator (4008)
Add AGC health check endpoint (3997)
Tolerate and require typed Werkzeug (4011)
Add more static URLs for Singularity debs (4007)

_Bug Fixes_

Update WES set up docs (4027)
Add real time logs (4031)
Fail fast if Docker builder is missing (4001)
Make Toil version be reported as a string in WES (4013)
Fix assorted typos within assorted comments (4023)
Make file store case insensitive (4153)
Pre-lex commands for qsub (4150)
Update Cactus and exclude broken networkx (4107)
Make `toil kill` work when the leader is on another machine (4084)
Wrong filename in output (4139)
Tolerate a missing VersionID key to fix 4129 (4130)
Only import from typing_extensions on old Python where we install it (4090)
Allow missing username and fix Docker build (4077)
Leave more time for concurrency measurement to fix 4012 (4068)
Stop people asking for ARM Mesos clusters to fix 4057 (4058)

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Hexotical, gmloose, kannon92, douglowe, gcapes, and pmiddend!

5.6.0

Changelog


Highlighted Features Added
* Integrate ARM Docker builds to make multi-arch images. (3802)
* WES support and server mode. (3779)
* TES batch system prototype. (3821)
* Support for new resource syntax in PBSPro. (3048)
* Toil now looks for lost jobs every minute instead of every hour (3948)

Breaking Changes
* Remove --disableCaching's true/false argument. (3869)

CWL
* CWL helper jobs: better disk & memory requirements. (3834)
* CWL: safer test path generation for post-install testing. (3818)
* More detailed names, which show up in job names sent to BatchSystem schedulers. 3941 (was 3893)
* If you use scatter and collect files,  duplicates are correctly dealt with and renamed. (3968)
* CWL: at the end of a job, ask cwltool to cleanup (3965)

Kubernetes
* Assign Kubernetes jobs an explicit TTL. (3936)
* Wait for node creation. (3934)

Dependencies
* Enable Dependabot updates. (3827)
* Multiple consolidated dependabot updates. (3851)
* Update addict requirement from <2.3,>=2.2.1 to >=2.2.1,<2.5. (3861)
* Bump cwltool to 3.1.20211107152837. (3833 3866 3909)
* Bump cwltest from 2.1.20210626101542 to 2.2.20210901154959. (3848)
* Bump flake8 from 3.8.4 to 4.0.1. (3847)
* Allow more docker-py versions. (3860)
* Remove pyyaml dependency. (3858)

Misc
* Update cleanup script. (3937)
* Remove use of sys.maxsize. (3824)
* Spelling fixes. (3814)
* Add codeql-analysis for Python. (3825)
* Update jobstore function names. (3809)
* Move AMI functions to lib. (3810)
* Add "make pyupgrade" (py36-plus). (3805)
* Type hints.  (3930)
* Coalesce status calls in slurm. (3822)
* Python logging takes format values as *args. (3852)
* Change quick test to a 10 minute timeout. (3843)
* Add make uninstall to makefile. (3883)
* Fix toil kill to find shared pid.log file (with unit test). 3941 (was 3932)
* Update documentation. (3947)
+ Remove remains of Travis. (3976)

_Bug Fixes_
* Stop checkpoints from being reissued multiple times. (3931)
* Don't consult LSF config when explicitly defining memory units. (3820)
* Robustly remove state dirs. (3836)
* Fix exception checking for exit_code. (3830)
* Use exitStatus instead of exitReason for batch exit type comparison. (3839)
* Update cwltest to improve K8 runs. (3935)
* Consolidated CI Fixes. (3887)
* Fix CWL conformance tests. (3891)
* Toil-managed cluster scaling should work again with `--metrics`. (3943)

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Hexotical, tmooney, nikhil, kannon92, douglowe, mhpopescu, Phhere, and gmloose!

5.5.0

Changelog

CWL
* Add podman support; and other fixes from recent cwltool 3799
* Add streaming feature for cwltoil 3694
* Warn users if a different cwltool version is installed 3686
* Turn on all Kubernetes CWL tests that are expected to work on Singularity 3720
* Fix CWL in toil docs jobstore usage 3728
* DOC: update versions of CWL support
* Allow filestore bypass 3652

Misc
* Numerous Type Hints. 3705 3701 3693 3691 3663 3688 3642 3684 3682 3680 3666 3675
* Single source of truth for job state 3776
* Do not set default for statePollingWait 3774
* Use absolute local paths when exportFile/importFile do not detect a schema 3767
* Multi-zone balancing within regions for AWS autoscaling groups 3746
* Migrate cloud-config to ignition 3488
* 🎡 Wheel Of Issues 🎰 3760
* Add back addBatchSystemFactory function 3754
* Redirect stderr to /dev/null of lsf conf queries 3751
* Set number of cores based on job.cores for OpenMP applications 3739
* Google jobstore batching 3740
* Locations of CLI option docs
* Add AWS provisioner storage system 3727
* Set cls.bucket 3726
* Stream vs dowload jobs 3722
* Update Toil's main python test version to 3.8. 3669
* Move Travis tests to Gitlab 3675

_Bug Fixes_
* Don't leak symlinks 3795
* Prevent exception from being raised when modifying dir permissions for clean up 3778
* Fix scontrol output parsing 3793
* Fix AttributeError 3742
* Workaround for S3 in us-east-1 3710
* Time data format 3708
* Fix leader.py batch system std files prefix glob 3679

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Hexotical, ionox0, gmloose, juanesarango, mhpopescu, mberacochea, nikhil!

5.4.0

Changelog

CWL
* Fix cwl and wdl dependency bleed and add stand alone tests. 3582
* Use MpiConfig.load() to handle MPI config file 3574
* Add support for MPI with CWL. 3525

Misc
* Numerous Type Hints. 3571 3634 3626 3625 3616 3614 3601 3592 3590 3581
* Configurable Grafana port 3597
* Handle streaming reads from the cache when the data isn't written back yet 3595
* Balance over pools using AWS ASGs 3490
* Change how Kubernetes schedules and scales for hopefully better scale-down behavior 3587
* Add Owner tags to our AWS buckets when testing. 3577
* Allow workdir override 3586
* Additional info for "Permission denied" error 3579
* Consolidate memory functions 3529
* Pen children 3482
* Sniff raid better 3526
* Add a propagation policy to batch delete 3522
* Cleanup script for buckets, sdb domains, instance profiles, and roles. 3373
* Add decorator for flaky tests. 3510

_Bug Fixes_
* Enforce valid uuid from getNodeId() 3611
* Removed extra memory multiplication 3608
* Fix non integers lsf memory requested 3609
* Fixes jobCommand unpack 3610
* Fix CWL test 20 on Kubernetes 3572
* Fix 3579 breaking AWS docs 3596
* Update cactus test to fix broken bucket links. 3594
* Allow cleaning up a job whose overlargeID fell off 3584
* Only deploy a user script if we can deploy user scripts 3518
* Reduce path name lengths. 3438
* Insist on credentials for testing AMI finding 3524

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, thiagogenez, julian-klode, darafferty, nikhil, mhpopescu!

5.3.0

Changelog

CWL
* Run CWL conformance tests on Kubernetes. 3323
* CWL symlinking files into work directory 3445
* set resource reqs for all CWL_INTERNAL_JOBS 3442

WDL
* Add WDL flatten() function + tests. 3485
* Add WDL collect_by_key() function + tests. 3476
* Add WDL keys() function + tests. 3460
* Add WDL as_pairs() function + tests. 3364
* Add as_map() WDL function 3448
* Add basic WDL development support 3434
* Add basic WDL 1.0 support 3421
* Update dictionary structure in AnalyzeWDL. 3416

Misc
* Add extra context for disk usage warning. 3495
* Update developing.rst 3479
* Allow text/encode options for read and write functions in file store 3428
* Type hinting and mypy checking additions 3470  3458 3456
* Quality of life improvements for cactus. 3463
* Add checklists for reviewing and merging PRs 3432
* Ensure that python3 and not python (2) is used 3446
* Support Kubernetes clusters in toil launch-cluster 3357
* Additional check for predecessors. 3417
* Allow setOptions to accept an argparse group. 3426

_Bug Fixes_
* AWS: Handle socket timeout during AWS discovery 3503
* Use back-up flatcar AMI if not found. 3513
* Lsf command parser 3475
* Retry on s3 throttling. 3504
* Stop deleting all ASGs with tags 3474
* Only determine execute permissions on files we copy/move. 3437
* cope with an invalid HTTP_PROXY 3447
* Fix failing Google jobstore test 3420
* Update Google job store 3412

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Arthur Rand, mberacochea, Jessime, thiagogenez, julian-klode!

5.2.0

Changelog

CWL
* Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111).  3375
* Add better handling for potentially mis-ordered CWL args. 3395
* Confirm CommandInputParameter expression can receive a File object. 3350

WDL
* Make AnalyzeWDL abstract to allow implementation for different WDL versions 3391

Misc
* Clean up memoize.py.  3374
* Refactor bioio library, simplify logging, and small adjustments to utils.  3351
* Use regular division for memory calculation on LSF. 3387
* Remove unused --nodeOptions arg. 3397
* Get boto S3 args for minio from environment variables. (3370)
* Update EC2 lists. 3376
* Add a retry to AWS destroy. 3379
* Regular division for memory calculation in LSF. 3387
* Add option to blank mem allocation for SLURM. 3399
* Preserve file permissions on imported/exported files 3322

_Bug Fixes_
* Preserve file permissions on imported/exported files. 3322
* Make building a flattened list of live jobs non-recursive. 3394
* Move AWS job store uploads/downloads to boto3 (reopened) 3400
* Use a Docker Hub mirror when pulling Docker images. 3411
* Corrected gridengine maxMEM check to use memorystring 3410

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, douglowe, stevekm , thiagogenez!

5.1.0

Changelog

WDL
* Add WDL cross function.  3360

_Bug Fixes_
* Better error message for insufficient disk.  3362
* Fix maxMEM comparison in LSF.  3369

Thank you to our contributors: mr-c , adamnovak , w-gao , jonathanxu18 !

5.0.0

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (3225) and replaced by:
- singleMachine -> single_machine
- gridEngine -> grid_engine
- LSF -> lsf
- Mesos -> mesos
- Slurm -> slurm
- Torque -> torque
- HTCondor -> htcondor
- Kubernetes -> kubernetes
- k8s -> kubernetes

Changelog

CWL
* Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). 3137
* Export to a bucket if cwl's CreateFile is specified. 3124
* CWL v1.1 Support: loadContents if True for StepValueFrom objects. 3266
* CWL: Fix workdir permissions. 3229
* Test CWL secondary file from s3. 3311
* CWL v1.1 Support: Inplace update has side effect on directory content. 3280
* More CWL v1.2 conformance tests. 3336
* CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. 3333

WDL
* Add ceil() and floor() WDL functions. 3168
* Create a stdout and stderr file for each WDL task. 3181
* Add WDL write functions and builtin tests. 3236
  - write_lines
  - write_tsv
  - write_json
  - write_map
* Add WDL read functions and builtin tests. 3244
  - read_lines
  - read_tsv
  - read_json
  - read_map
  - read_int
  - read_string
  - read_float
  - read_boolean
* Add transpose() WDL function. 3273
* Add length() WDL function. 3307
* Add sub() WDL function. 3252
* Add range() WDL function. 3277
* Add size() WDL function. 3255
* Add zip() WDL function. 3355
* Implement WDL Pair type. 3304
* Refactor WDL types 3335

Kubernetes
* Adding Labels to Pods/Jobs. 3233
* Make kubernetes & botocore packages fully optional. 3259 3261
* KubeWatch optimization. 3227
* Catch other kubernetes imports. 3261
* Automatically infer singularity for kubernetes. 3279

Misc

* Allow Caching By Default. 3111
* Update dependencies and loosen restrictions slightly. 3139
* Ensure that falsey (but non-null) values are considered. 3143
* Remove pathlib. 3169
* Move AWS job store uploads/downloads to boto3. 3153
* Update prom/node-exporter to use quay.io. 3186
* Add support for TOIL_CUSTOM_INIT_COMMAND. 3183
* Update docker module version to 4.3.1. 3222
* Add 'stream' and 'demux' options to apiDockerCall. 3224
* Use boto3 to create our leader node. 3145
* Prevent job kind filenames from getting too long. 3230
* Add a retry decorator. 3144 
* Remove mesos offer messages. 3308
* Job Concept Unification. 3250
* Add reporting for accessed files of failed jobs. 3309
* Add coverage and refactor. 3314
* Expose --statePollingWait param through toil. 3321
* Record the maximum memory used by LSF jobs. 3327
* Remove custom math functions and replace with built ins. 3338
* Define a changelog process. 3316
* SLURM: getting job details from `scontrol show jobs` was non-functional and has now been fixed. This behavior in the case of `sacct` not being configured now works. 3346
* LSF: handle unicode configuration files 3354
* Replace abssympath w/ builtin. 3356

_Bug Fixes_
* Turn off watch code in Kubernetes. 3175
* Check against joining the current thread during CachingFileStore destructor. 3187
* Slurm.py error, undefined stdout - stdout should be stderr in code. 3194
* Fix restart flag error (issue 3094). 3142
* Batch System generates a key error instead of a usage message. 3225
* Remove Python 2 imports in WDL. 3228
* Move checkForDeadlocks() in the leader. 3234
* Fix boto3 migration bugs. 3243
* Fix None check when parsing WDL JSON file. 3272
* Fix broken stdout/stderr in torque job. 3306
* Add doubleMem for LSF jobs that die due to imposed memory limit. 3313
* Add 500 status to the list of retriable SDB BotoServerErrors. 3329
* Add a retry logic for BucketNotEmpty error. 3341
* Fix None check when parsing WDL JSON file. 3272
* Warn when trying to clean non-existent job store 3288
* Only rm a statefile that exists. 3348
* CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, w-gao, jonathanxu18, jeffrey856, davidlougheed, altairwei, mberacochea, drkennetz, ionox0!

4.2.0

* Don't log in the inner scheduling loop. 3065
* Handle anticipated 404 errors without retrying. 3067
* Test and reimplement AMI finding via Flatcar JSON feed. 3061
* Bump Enlighten version to partly fix 3069. 3070
* Get wait duration is far too short of a rest for lsf. 3076
* Unify logging format. 3073
* Support CWL 1.1 Listing. 3058
* Enable CWL v1.2.0-dev3 (3.0.20200324120055 -> 3.0.20200530110633). 3092
* Enable CWL v1.2.0-dev4 (3.0.20200530110633 -> 3.0.20200709181526). 3105
* Add --nodeStorageOverrides option. 3096
* More robust CWL filepaths and secondary files. 3114
* Replace custom mkdir_p with the built in. 3123
* Move conditional execution test to before fill_in_defaults. 3117

_Bug Fixes_
* Stop CWL from trying to copy downloaded directories from somewhere local. 3053
* Exit the worker with 1 when jobs fail so the batch system can see. 3052
* Eliminate recursion in root finding to fix 3080. 3081
* Add a Cactus Kubernetes test to the integration tests on Gitlab. 3077
* Chaining doesn't make sense for CWL. 3091
* Fix failed job accounting. 3103
* Improve lsf batchsystem stability. 3101
* Deduplicate updated jobs. 3108
* Redundant virtualenv check. 3135
* Change Slurm command output processing. 3133
* Fix for log file being linked more than once. 3130
* Wrap Kubernetes watches to fix 3125. 3126
* Create dirs that do not exist when bind-mounting to docker. 3120
* Fix database state when file upload fails. 3122

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans, nikhil, drkennetz, jonathanxu18, tobiaszjarosiewicz, ionox0!

4.1.0

* Attempt to make batch system more robust and debuggable. 2959
* Kubernetes Shared Caching. 3012
* Test python3.8 and build a python3.8 appliance. 3028
* Use the original pymesos. 3036
* Detect more kinds of virtual environment and normalize prefixes. 3035
* Do More Hashing.  Can't have enough. 3025
* Replace reissued jobs message with better progress indicators. 3044
* Detect insufficient resources and deadlocks more usefully. 3043

_Bug Fixes_
* Don't list specific CWL features, try them all (and fix some CWL running bugs). 3015
* Always honor TOIL_GRIDENGINE_PE and never assume a site-dependent default. 3041

Thank you to our contributors: mr-c, arostamianfar, adamnovak, diekhans!

4.0.0

This release moves to python3 only, dropping all python2.7 compatibility, and also deprecates the command "cwltoil" in favor of "toil-cwl-runner".

* Update cwltool version (==1.0.20190906054215 -> <=2.0.20200126090152). 2969
* Drop the long-deprecated cwltoil. 2843
* Dropped python2.7. 2973
* Port provisioner tests and only test py3.6. 2842
* CWL: Refactor of link merge + conditionals + pickValue. 2845
* Add import subprocess to executor.py. 2976
* Encode mtail stdin to utf-8 and flush. 2978
* Encode framework message as utf-8 as well. 2981
* Make the output files for grid engine batch systems not try and be in per-host directories. 2956
* Use file locks instead of PID polling to see if other processes sharing the cache are alive. 2982
* Size downloads during download. 2989
* Add timeouts to all SQLite connect statements. 2993
* Eliminate sleep time on --restart. 2990
* Add support for --awsEc2ExtraSecurityGroupId. 2997
* Add moveExports option. 2983
* Set worker threads as dameon to prevent hanging process on error. 2998
* Cut the Threads. 2999
* [Part 1] CWL v1.1 Support 2985
* [Part 2] CWL v1.1 Support 3000
* Update Sphinx and let its version float. 3011
* Add --enableUnlimitedPreemptableRetries option. 2896 
* Add less and vim to docker. 3017
* Improve and Unify Log Dumping. 3008
* Make dependency on Python3.6+ machine-readable in setup.py. 3021
* Improve slurm update frequency. 3026
* Make Sphinx a make prepare component and not a dependency. 3023
* Use forked pymesos and http-parser dependencies. 3024

_Bug Fixes_
* Fix python3 string/bytes error when using --printLogs. 3005
* Fix connection between Kubernetes and Leader. 3004

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856, kaushik-work, glmxndr, johnbradley, cmarkello, diekhans!

3.24.0

This is the last version of Toil that supports python2.7 and allows the command word "cwltoil" (now "toil-cwl-runner").

* Update ssh and docker setup files. 2958
* Update EC2 instance types. 2962
* Source toil environment as a configuration file. 2960
* Scan for the latest caching database instead of using a hard link. 2948
* Specify py3.6 or py3.x instead of just py3 in appliance. 2966

_Bug Fixes_
* Fix database being locked when trying to write in the CachingFileStore. 2955
* Fix py3 spot market scaling. 2971

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856!

3.23.1

Functionally the same as Toil Version 3.22.0.

3.22.0

* Make an appliance for Python 3. 2937
* Use status to properly get the HTTP status code from boto. 2934
* Use atomic file copy when writing to posix file system. 2912
* Improve Kubernetes job timing. 2922
* Increase SQLite DB timeout to 60s. 2913
* Consider pods that are very close to their memory limits to be stuck. 2895
* Constantly reload Kubernetes config. 2869
* Up the disk to the default as well as the memory. 2880
* Tolerate unnecessary bucket cleanup. 2894
* Add checksum verification for AWS job store. 2889
* Change python3.6 ppa to deadsnakes. 2888
* Change Kubernetes job sizing logic. 2877
* Added env var TOIL_KUBERNETES_OWNER. 2872
* Raise job limits to be more than their requests. 2868
* Remove all Azure support from toil. 2860

_Bug Fixes_
* Exclude .nfs files from deferred function search. 2930
* Fix cpu_count() to actually read quotas. 2933
* Compile http-parser so that it doesn't break pip dependencies. 2940
* Use an http-parser for pymesos that builds on 3.7+. 2931
* Fix botocredAdapter. 2904
* Make sure to commit when giving away files. 2914
* Actually call the implemented size polling method. 2916
* Set Docker vars back to ?=. 2908
* LSF: Don't check for Status <EXIT>, as that breaks the error code finding. 2863

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856, diekhans!

3.21.0

* Add kubernetes watch. 2848
* Add service account configuration for Kubernetes. 2851
* Head object 404 with encryption. 2853
* Tests for Kubernetes. 2838
* Turn usernames into non-spooky Kubernetes name components. 2836
* Make examples a symlink to make examples discoverable. 2834
* Kubernetes BatchSystem Support. 2786
* Specify GCE coreos image by family name instead of specific version. 2829
* Add toil-cwl-runner --provenance. 2785
* Revise caching more. 2770
* Bump psutil >= 3.0.1, <6. 2790
* Add "failed_" prefix to failed logs. 2792
* Add --writeLogsFromAllJobs option. 2781
* Expose --default-container to toil-cwl-runner. 2784
* Update cwltool from 1.0.20190228155703 -> 1.0.20190815141648. 2764
* Pass jobID to Torque Wrapper. 2762
* Limit Length of Log Names. 2755
* Add CWL options --strict-memory-limit, --relax-path-checks, and --leave-container. 2718
* Update EC2 nodes and script. 2709
* Redesign File Job Store layout. 2669

_Bug Fixes_
* Fix for passing default values between Workflows. 2831
* Ignore checking versioning consistency when cleaning s3 jobstore. 2826
* Ensure terminated ignored nodes are cleared from batch system. 2789
* Fix Slurm sbatch command. 2777
* Typo in gridengine.py. 2778
* CWL fixes. 2783
* Clarify `/data` mountpoint for `dockerCall`. 2747
* Fix CWL tmpdir to be unique per job. 2745
* Protect against potential AWS truncated downloads. 2731
* Fix runningOnEC2 to work with all instances. 2734
* Prevent Toil from zipping entire dirs. 2728
* Changes to lsf.py to make it compatible on lsf9 and lsf10. 2726
* Remove X86_64 type requirement from lsf support. 2723
* Clean up temporary directories cwltoil uses to hold output streams. 2690

Thank you to our contributors: mr-c, arostamianfar, adamnovak, jeffrey856, tthyer, kaushik-work, dleehr, ionox0, psafont, mb1069, tobiaszjarosiewicz, gconcepcion, byee4, smoe, TMiguelT, jvivian-atreca!

3.20.0

* Deprecate subprocessDockerCall(). 2653
* Move wallTime to a label so we can use exact timestamps. 2664
* Capture and log standard output/error from batch systems. 2662
* Add an option to set custom AWS IAM roles. 2656
* Add a ToilNodeType tag to distinguish leader vs. workers. 2658
* Slurm: include both exit code and terminating signal in job return status. 2646
* Update requests package. 2610
* HTCondorBatchSystem(): fixes and features. 2647
* Update cwltool from 1.0.20180820141117 -> 1.0.20190228155703 2609
* Add an option to set disableCaching to false. 2584
* Speed up S3 bucket deletion. 2597
* Tolerate missing jobs/workers in stats expando. 2533
* Add private IP support to node operations. 2569
* Encode logs for gzip. 2595

_Bug Fixes_
* Improve NoSuchFileException usage and robustness. 2593
* Make sure we always have current package lists. 2675
* Make sure to use --tmpdir-prefix to place workdir and jobstore. 2587
* Subprocess dockercall bytestring compatibility for py2/3. 2677
* Address python3 deprecation warnings. 2668
* Fix python3.7 break on __new__ static method. 2659
* Get value of taskId in Mesos' killTask method. 2661
* Stop NamedTemporaryFile tampering. 2638
* StatsAndLogging(): ensure json.load() receives only string input. 2641
* Reduce max size of files allowed to be inlined in SDB. 2440
* Increase hardcoded encapsulate follow-on resources. 2591
* Ensure TOIL_DOCKER_REGISTRY variable is in version_template. 2583
* Use current python executable. 2599

Thank you to our contributors: mr-c, arostamianfar, adamnovak, kwwette, liinus, saimeCS, cdunn2001, Fifan31, jeffrey856!

3.19.0

* Support for CWL s3 imports. 2522
* Support for running custom init commands in docker (allowing auth in private dockers). 2561
* Add --no-read-only option to the CWL runner. 2531
* Add support for assuming roles. 2527
* Added an adapter to allow Boto 2 to use credentials obtained via Boto 3. 2527
* Add a flag for running CWL internal jobs on workers. 2524
* Added CWL tests for http, https, and s3 file imports. 2234
* Begin testing python3 in gitlab.  2518
* Relegate pytest dependency to the dev reqs.  2508
* WDL changes to enable resource usage. 2495
* Sphinx and pathlib2 dependencies. 2490 2493
* Upgrade appliance to Ubuntu 16.04. 2480
* Update cwltool from 1.0.20180820141117 -> 1.0.20181118133959 2469
* Add CWL's --preserve-entire-environment option. 2481
* Bump minimum version of galaxy-lib (17.9.3 -> 17.9.9). 2476
* Added all tutorial scripts to unit testing. 2426
* More python3 support added. 2405
* Update the CoreOS AMI. 2430
* Simplify log creation in leader. 2423
* Support for hardlinks and greater testing for toil status. 2402
* Pymesos port. 2366 2338
* Use displayName for CWLJob to show stats in toil --stats. 2397

_Bug Fixes_
* Import errors. 2558
* Fix boto3 credentials for permanent credentials. 2549
* Fix bug in Toil stats. 2513
* Replace some str objects as real FileIDs. 2500
* Remove linear-time bottleneck from Mesos batch system. 2498
* Slurm and py3 fixes. 2494
* Safeguard unintentional boto imports. 2489
* Fix broken readthedocs build. 2485
* Specify region in prometheus. 2437
* Fixed bytestring-string tagging. 2472
* LSF log job id on error. 2450
* Fix massive slowdown when many jobs are running. 2439
* Make ServiceManager stop blocking while (most) services are established. 2441
* Default to bash on the leader node rather than sh. 2435
* resource.getusage returns KB already, so dividing by 1024 yields MB. 2425
* Recognize conda virtual envs. 2422

Thank you to our contributors: mr-c, arostamianfar, adamnovak, nikhil, lamby, gijzelaerr, joelarmstrong, markasbach, davidjsherman, psafont, jsmedmar, bencvdb, emlys, ionox0, ibebio, MarkFilus!

3.18.0

* Refactor stream writes in the fileJobStore to minimize holding open file handles
* Large python 2/3 compatibility changes (+ python 3 compatible mesos library: pymesos)
* Get status/PID API added
* CWL:fix support for user space overrides
* Update cwltool from 1.0.20180518123035 to 1.0.20180820141117
* support lsf 10 per core reservation
* WDL bucket outputs with orig names
* Grid engine job creation fix
* Add Mesos port forwarding and the ability to use a SOCKS proxy with toil ssh-cluster
* Require a jobstore to be specified when a provisioner is.
* Retry HTTP connections on BadStatusLine errors
* Quote() all Docker arguments
* log LSF job id to error on failure
* Improve test file cleanup
* retries for batch commands
* Update CoreOS AMI
* CWL workDir defaults to tmpdir_prefix now if not set by the user
* Set the default duration for batch systems with getWaitDuration()
* Coverage Evaluation

_Bug Fixes_
* AMI Fetch Timeout
* Stop logging every message twice in certain cases
* Fix htcondor in wheel breaking on Apple
* Fix toil kill
* Fix invalid qsub in torque.py

Thank you to our contributors: mr-c (big thanks for all of the CWL support!), MarkFilus, xzhub, ionox0, jsmedmar, bencvdb, tom-dyar, joelarmstrong, mwiens91, jessebrennan, and adamnovak!

3.17.0

* Add CWL destBucket runtime option to deposit CWL outputs into cloud buckets
* Update boto3 and boto
* Better Error messages when docker images are unavailable
* Broader checking for docker image existence prior to launching blindly
* Incremental Python2/3 compatibility improvements
* Enable support for specifying zone in all environment variables
* Expose `--force-docker-pull` and `--no-match-user` options from cwltool
* Add mem free option to UGE
* Convert most log messages to debug level
* Update cwltool from 1.0.20180518123035 to 1.0.20180521150620
* Incremental WDL Support Improvements (docker permissions, parsing, and improved logging)
* Documentation Housekeeping and Code Cleanup

_Bug Fixes_
* Stop FileJobStore DEBUG messages from sneaking out once for every job
* WDL Parsing Error
* Fix docker daemon detection depending on terminal size
* Checking virtualenv
* Properly filtering AWS security groups
* Fix a crash if job is larger than all available node types
* Avoid overwriting NVMe root volumes on current-gen AWS instances
* Don't call addNodes if the delta was met by unignoring nodes
* Fix AWSJobStore updates not working if values are shrunk
* Prevent worker from checking for nonexistant file


Thank you to our contributors: mr-c (big thanks for all of the CWL support!), jsmedmar, bencvdb, tom-dyar, joelarmstrong, mwiens91, jessebrennan, and adamnovak!

3.16.0

* More Robust WDL Support
* Reworked the Cluster Scaler Algorithm
* Updated subprocess32 to version 3.5.1
* Eliminated bd2k-lib dependencies
* apiDockerCall allows blocking now for both detached/attached images
* Util to fetch the latest AWS instance types
* Check if docker images specified by the Toil appliance are valid before running
* Reduce toil clean memory usage for AWS jobStores

_Bug Fixes_
* CWL fixes for handling promises / followOn jobs
* Fix occasional cPickle deadlocks
* Increased Google and Azure stability
* Google jobstore fixes
* fileJobStore deletion made more robust for NFS-like systems

Thank you to our external contributors: mr-c (big thanks for all of the CWL support!), chapmanb, ryanking, jmchilton, jorvis, evan-wehi, brainstorm, jsmedmar

3.15.0

* Azure Support (Beta)
* Google Support (Beta)

_Bug Fixes_
* CWL fixes.
* LSF cpu count fix.

Thank you to our external contributors: mr-c (big thanks for all of the CWL support!), chapmanb, ryanking, jmchilton, jorvis, evan-wehi, brainstorm, jsmedmar

3.14.0

* Added support for the HTCondor Batch System
* Updated documentation.
* Added the protobuf dependency for mesos.
* Updates for CWL (scheduling, update CWL ref runner, update conformance tests, expose cwltool's tempdir-prefix, expose user-space-docker-cmd, and always import links for CWL)
* Added an "all" option to install all extras.
* Check for venv in python 2 and 3.

_Bug Fixes_
* Added buffering for local file transfers to fix MemoryError bugs
* Fix race conditions for SLURM, GridEngine, Mesos, and Torque batch systems.
* Updated file size determination on Apple products.
* Fixed a bug when making the docs through Sphinx.
* Amended memory limit and reservation in LSF.
* Removed cwl-runner command, as it was interfering with the native CWL command.

Thank you to our external contributors: mr-c (big thanks for all of the CWL support!), chapmanb, ryanking, jmchilton, jorvis, evan-wehi, brainstorm, jsmedmar

releases/3.13.0-dirty
* WDL Alpha support
* Job store symlinking
* Updated cwltool support (version; fixes; testing)
* Better logging
* Option to get stderr from subprocessDockerCall
* Integration testing with ENCODE
* Cactus with autoscaling


_Bug Fixes_
* Fixed static provisioning
* Fixed launch-cluster bug when terminal size was too small
* Removed AWS specific code from the abstract provisioner

Thank you to our external contributors: mr-c, chapmanb, ryanking, jmchilton, jorvis, evan-wehi, brainstorm, jsmedmar

3.12.0

* Forkless mode for debugging
* Enabled Travis builds with a subset of tests (less than 15 minutes).
* DockerCall function changed to use docker API.
* Updated to Azure version 2.0.
* Added tempDir convenience function to job and made job.log an alias for fileStore.LogToMaster.
* Enhanced toil-cwl-runner to support SoftwareRequirements & BioContainers.
* Increased default retry count to 1.
* Deprecated cgcloud.
* Updated EC2 instance types.
* Upgraded pip to 9.0.1 and setup tools to 36.5.0 in appliance build.

*Bug fixes*
* Parasol parses jobStore path correctly.
* Parasol doesn't attempt to remove jobs that it didn't launch.
* Fixed problem with dynamic batch systems not always being created.
* Fix workflow effort being wasted if a single export fails.
* Added a default value for maxNodes.
* Fixed py.test collection errors when running locally.

Thank you to our external contributors: jmchilton, jorvis, evan-wehi, mr-c, brainstorm

3.11.0

* Allow batch creation of jobs.
* CWL: run internal jobs locally, not on cluster.
* TOIL_TEST_QUICK environment variable for running a subset of tests.
* Allow clusters with multiple node types.
* Put jobName into the _toil_worker arguments.
* GridEngine: cache instead of block for statePollingWait

_Bug Fixes_
* CWL: avoid early removal of the temporary outdir
* Fixed AWS readFileStream deadlock on partial reads.
* Use a socket to get the public-facing IP

Thank you to our external contributors: chapmanb, ryanking, mr-c

3.10.1

Adds missed commits that should have been in 3.10.0.

3.10.0

* Full CWL 1.0.1 support.
* Improved documentation with better examples.
* Added option to link files when importing into fileJobStore.
* Allowed some interactive support
* Improved local testing targets.
* Sleep support for all state-querying functions in batch.
* Opened up UDP traffic within AWS clusters.
* Set mesosMasterAddress default to private IP.
* CWL: symlink jobStore files instead of copying.

_Bug Fixes_
* Fixed getEmptyFileStoreID.
* Automatically use the correct LSF units.
* Fixed failure in resolve_indirect_inner in cwltoil.py.
* Fixed docker tests failing on Mac.
* Stopped log level option defaults from clobbering each other.
* Various other minor bug fixes.

Thank you to our external contributors:
mr-c, tetron, liuxf09, ryanking, OliverEvans96, alaindomissy, brainstrom, chapmanb, nazamy, jmchilton, stefanoberri

3.9.0

* Added a plugin batch systems that abstracts option settings for batch systems.
* Fixed mysterious fail with rsync `--protect-args` on OS-X.
* Added environment variable TOIL_AWS_KEYNAME for setting AWS credentials during tests.
* Tests now timeout with pytest rather than Jenkins.
* Added support for PBSPro (Torque's proprietary version).
* Fixed resolution of symlinks for hard links bug.
* Added an option to set root volume on leader and worker nodes.
* Fixed autoscaler crash.
* Added environment variable TOIL_TORQUE_ARGS for setting Torque arguments.
* Fixed hot-deployment with python-3-supporting libraries.
* Fixed race in ClusterScaler test.
* Several logging improvements and fixes.
* Pip show now displays license info.

3.8.0

- Don't error if job hasn't been submitted yet (resolves 1666) 
- Match doc to reality for --preserve-environment 
- CWL: symlink jobStore files instead of copying 
- When killing tasks, make sure to reap child processes in the process groups 
- Better error checking in piped docker commands 
- Fix case where a job has been put in currentjobs, but not yet dispatc… 
- Use hard links instead of copied files on import to file job store 
- CWL: Add support for InitalWorkDirRequirement 
- Set proper value for service job preemptable requirement 
- Use chunked transfer in AWS job store importing 
- Lower timeout for domain binding in Toil Clean 
- Add `import os`.

3.7.0

* Updated documentation at http://toil.readthedocs.io/en/latest/
* Improved Azure support
* Remove upper limit on size of jobs in AWS job store
* CWL workflows can now take advantage of Toil Autoscaling
* CWLToil bug fixes and enhancements
* Toil Appliance updated to Mesos version 1.0.1
* Improve scalability of Mesos batch system
* Better error message for unfulfilled promises
* Fix handling of checkpointed jobs
* Fix service job deadlock detection
* Allow disabling of hot deployment mechanism

3.6.0

- New Toil landing page at http://bd2kgenomics.github.io/toil/
- Tags all Toil-provisioned instances with an 'Owner' tag to improve cost tracking 
- Allows users to arbitrarily tag their Toil-provisioned instances
- Adds a new Torque batch system class
- Improved documentation
- Support for piping commands within DockerCall
- Improved cluster usage statistics
- Various bug fixes improving support for large file exporting, the Slurm batch system, and general Toil robustness

3.5.2

Hotfix release for 3.5.x 
- Fixes job state discovery race 
- Fixes race between job store file exporting and caching
- Fixes bugs in Slurm batch system

3.5.1

Hotfix release for 3.5.x
- Fixes import error raised when 'AWS' extra not installed

3.5.0

- Toil now includes a native EC2 provisioner for launching and controlling Toil clusters.
- Toil now has the ability to autoscale - if enabled, Toil will increase or decrease the number of worker nodes in its cluster depending on the current demands of the workflow
- Log output now refers to each Job by name.

3.4.0a1.dev228

This pre-release of Toil contains bug fixes and the addition of experimental, CGCloud-based auto-scaling.

https://github.com/BD2KGenomics/toil/milestone/17?closed=1

Due to the experimental nature of the auto-scaling feature, we decided to skip publishing a stable release for it. To install this release with pip, use `pip install toil==3.4.0a1.dev228`. 

The next stable release of Toil will contain docker-based autoscaling for EC2.

3.3.4

https://pypi.python.org/pypi?:action=display&name=toil&version=3.3.4
- Fix: Toil fails completely when not installed with [aws] extra (1115)
- Prevent virtualenv from silently upgrading pip and setuptools, potentially breaking build (1270)

Core production is advised to skip this release.

3.3.3

- Fix `--restart` not working at all on distributed clusters (resolves 1127)
- Work around apparent eventual consistency with SDB (resolves 760)

This release is recommended to replace 3.3.1 and 3.3.0 for production use.

[CGCloud 1.5.5](https://github.com/BD2KGenomics/cgcloud/releases/tag/releases%2F1.5.5) was released to deploy this release of Toil on toil-box images/instances.

The 3.3.2 release was invalidated due a error late in the release process.

3.3.1

This hotfix release adds disk tracking to the single machine batch system (770). Previously, that batch system ignored any disk requirements associated with jobs. The fix was contributed by arkal.

3.3.0

This major release adds bug fixes and a minor feature, specifically:
- `toil clean` is more robust and can clean partially corrupt job stores (869 et al)
- `job.fileStore.writeGlobalFIle(…)` returns a magic string with a `.size` attribute reflecting the size of the file just having been written. In conjunction with promised requirements, this small change greatly simplifies dynamic computation of job requirements as a function of the size of another job's output (1014)
- Two race conditions in the caching layer are fixed (1046 and 1033)
- A show-stopping race condition with promised requirements was fixed (1039)
- Several other bug fixes and code quality improvements
- The undocumented `cache=` job requirement has been removed

3.2.1

This major release adds many new features, improvements and bug fixes. It is the new stable release and users are encouraged to upgrade to it. Support for the previous stable release, now considered legacy, will continue but be limited to security fixes.

The important new features are:
- Completely rewritten caching allows for sharing of files between worker processes running on the same worker node. Previously, only a chain of jobs running inside a single worker process would benefit from caching.
- Improved ability to run long-running jobs that provide _services_ to other, regular jobs in the workflow.
- Building on top of the improved service jobs feature, we added support for ephemeral Spark sub-clusters that are formed on-demand during a Toil workflow and can be used by jobs to perform ADAM or other Spark workloads.
- Support for hosting job stores in Google Cloud Storage was added. Install Toil with the `google` extra to enable this support. The Google job store should be considered experimental. The Toil team is working on gaining more experience on the consistency and atomicity guarantees provided by Google Cloud Storage to ready that job store implementation for production use.
- Support for hosting job stores in Azure Storage was demoted to experimental status due to atomicity and consistency issues. The Toil team will be working closely with Microsoft engineers to resolve these.
- Added ability to import and export files into a job store on both the leader prior/after to invoking a workflow and on the workers as part of a workflow. Importing files from `http://`, `https://` and `file://` URLs is supported out of the box. So is exporting to `file://` URLs. Installing Toil with any or all of the `azure`, `aws` or `google` extras extends the import/export facility to files hosted on Azure Blob Storage, S3 and Google Cloud Storage, respectively.
- Toil scripts can now have dependencies and those dependencies are deployed automatically to each worker node. The script and the dependencies must be installed in a virtualenv and must be pure-Python. Native dependencies must be installed by the user on each worker prior to running a workflow. The Toil team uses the CGCloud project to handle those native dependencies.
- CWL support has matured.

3.2.0

This release contains critical bugs (1005 and 1007) and should not be used.

3.1.6

- Fix: User scripts fail when installed in a virtualenv that was created with --system-site-packages (929)
- Fix: Job functions fail to load if they are spread across more than one module (930, 831)

3.1.3

Changes:
- Fix rare bugs in Mesos scheduler (674, 672)
- Fix occasionalNoSuchJobException during --restart (639)

3.1.2

Changes
- Fix: Occasional KeyError during --restart (582, Christopher "CJ" Ketchum)
- Fix: Chained jobs overwrite each other's statistics with --stats (624, Christopher "CJ" Ketchum)
- Improve performance of --restart (Adam Novak)
- Fix: Mesos batch system doesn't favor bigger tasks (624, Hannes Schmidt)
- Various CWL integration fixes (Peter Amstutz, Kyle Elrott)
- Fix: Permission denied when adding files to job store that are not owned by current user (Adam Novak)

Links

• PyPI: https://pypi.org/project/toil
• Changelog: https://data.safetycli.com/changelogs/toil/
• Repo: https://github.com/DataBiosphere/toil
• Docs: https://pythonhosted.org/toil/

Update pip from 24.1.1 to 24.1.1.

Changelog

24.1.1

===================

Bug Fixes
---------

- Actually use system trust stores when the truststore feature is enabled.

Vendored Libraries
------------------

- Upgrade requests to 2.32.3

24.1

=================

Vendored Libraries
------------------

- Upgrade truststore to 0.9.1.

24.1b2

===================

Features
--------

- Report informative messages about invalid requirements. (`12713 <https://github.com/pypa/pip/issues/12713>`_)

Bug Fixes
---------

- Eagerly import the self version check logic to avoid crashes while upgrading or downgrading pip at the same time. (`12675 <https://github.com/pypa/pip/issues/12675>`_)
- Accommodate for mismatches between different sources of truth for extra names, for packages generated by ``setuptools``. (`12688 <https://github.com/pypa/pip/issues/12688>`_)
- Accommodate for development versions of CPython ending in ``+`` in the version string. (`12691 <https://github.com/pypa/pip/issues/12691>`_)

Vendored Libraries
------------------

- Upgrade packaging to 24.1
- Upgrade requests to 2.32.0
- Remove vendored colorama
- Remove vendored six
- Remove vendored webencodings
- Remove vendored charset_normalizer

``requests`` provides optional character detection support on some APIs when processing ambiguous bytes. This isn't relevant for pip to function and we're able to remove it due to recent upstream changes.

24.1b1

===================

Deprecations and Removals
-------------------------

- Drop support for EOL Python 3.7. (`11934 <https://github.com/pypa/pip/issues/11934>`_)
- Remove support for legacy versions and dependency specifiers.

Packages with non standard-compliant versions or dependency specifiers are now ignored by the resolver.
Already installed packages with non standard-compliant versions or dependency specifiers
must be uninstalled before upgrading them. (`12063 <https://github.com/pypa/pip/issues/12063>`_)

Features
--------

- Improve performance of resolution of large dependency trees, with more caching. (`12453 <https://github.com/pypa/pip/issues/12453>`_)
- Further improve resolution performance of large dependency trees, by caching hash calculations. (`12657 <https://github.com/pypa/pip/issues/12657>`_)
- Reduce startup time of commands (e.g. show, freeze) that do not access the network by 15-30%. (`4768 <https://github.com/pypa/pip/issues/4768>`_)
- Reword and improve presentation of uninstallation errors. (`10421 <https://github.com/pypa/pip/issues/10421>`_)
- Add a 'raw' progress_bar type for simple and parsable download progress reports (`11508 <https://github.com/pypa/pip/issues/11508>`_)
- ``pip list`` no longer performs the pip version check unless ``--outdated`` or ``--uptodate`` is given. (`11677 <https://github.com/pypa/pip/issues/11677>`_)
- Use the ``data_filter`` when extracting tarballs, if it's available. (`12111 <https://github.com/pypa/pip/issues/12111>`_)
- Display the Project-URL value under key "Home-page" in ``pip show`` when the Home-Page metadata field is not set.

The Project-URL key detection is case-insensitive, and ignores any dashes and underscores. (`11221 <https://github.com/pypa/pip/issues/11221>`_)

Bug Fixes
---------

- Ensure ``-vv`` gets passed to any ``pip install`` build environment subprocesses. (`12577 &l

[pyup-bot]

Closing this in favor of #52