Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

execute batch downloads in parallel worker threads #12923

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

execute batch downloads in parallel worker threads #12923

wants to merge 3 commits into from

Commits on Aug 20, 2024

  1. improve pooled progress output for BatchDownloader 

    - use more specific types for BatchDownloader#__call__
- calculate byte lengths with a HEAD request
- quiet all progress output from -q
- don't write colored output with --no-color
- write a lot more documentation for the new progress bar logic
- use ProgressBarType enum for --progress-bar CLI flag
    @cosmicexplorer
    cosmicexplorer committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    549781b View commit details
    Browse the repository at this point in the history

  2. download larger files first

    @cosmicexplorer
    cosmicexplorer committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    11a7a0f View commit details
    Browse the repository at this point in the history

  3. execute batch downloads in parallel worker threads 

    - limit downloads to 10 at a time instead of starting all at once
- add cli arg to limit download parallelism
- factor out receiving thread exceptions into a contextmanager
- default batch parallelism to 10
- make batch download parallelism 1 in html index test
- explicitly yield threads to help ensure correct download ordering
    @cosmicexplorer
    cosmicexplorer committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    6eeb26f View commit details
    Browse the repository at this point in the history