How to make large downloads work?

[markbulk]

I'm trying to download data starting from the beginning of 2018. The "Getting activities" process completes without issue. The first two monitoring data downloads work, but then it gets stuck. I assume that this is likely due to Garmin limiting pings on the API? Does anybody have a way around this as described below (largely following the bug template)?

Steps to reproduce the behavior:

1. Edit GarminConnectConfig.json to have start_dates of '01/01/2018'
2. Run sh download_create_dbs.sh
3. Wait for downloads to complete. Smoothly downloads for a long time, but then just stops. Replicated behavior twice.
4. See text below
5. I narrowed the date_starts in the config file to just last 7 months and was able to successfully complete the download and use the data.
6. Then edited the config file to push the start_date's back six months and then ran download_update_dbs.sh, but it only refreshed on the most recent information. This is probably reasonable given the design, but I don't yet know how I can fill in my full history.

Expected behavior
A complete download of all of my data from the start of 2018.

This is the best representation, I think that I can show:

sh download_create_dbs.sh
Creating new DBs by downloading from Garmin Connect
___Downloading All Data___
Geting activities: '/Users/markbulkeley/HealthData/FitFiles/Activities' (1000) temp /var/folders/9p/5j3w5chd2410q585389mpw_40000gn/T/tmpiut49exu
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [24:02<00:00,  1.44s/activities]
Downloading all monitoring data from: 2018-01-01 [1139]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1139/1139 [22:18<00:00,  1.18s/days]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1139/1139 [22:41<00:00,  1.20s/days]
 55%|███████████████████████████████████████████████████████████████████████████████▉                                                                 | 628/1140 [21:56<23:22,  2.74s/days] 55%|████████████████████████████████████████████████████████████████████████████▌                                                              | 628/1140 [12:34:46<10:15:21, 72.11s/days]
Traceback (most recent call last):
  File "garmin.py", line 341, in <module>
  File "garmin.py", line 322, in main
  File "garmin.py", line 135, in download_data
  File "download_garmin.py", line 225, in get_monitoring
  File "download_garmin.py", line 215, in __get_monitoring_day
  File "utilities/rest_client.py", line 166, in download_binary_file
  File "utilities/rest_client.py", line 137, in __download_file
  File "utilities/rest_client.py", line 104, in get
  File "requests/sessions.py", line 555, in get
  File "requests/sessions.py", line 542, in request
  File "requests/sessions.py", line 655, in send
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 672, in urlopen
  File "urllib3/connectionpool.py", line 421, in _make_request
  File "<string>", line 3, in raise_from
  File "urllib3/connectionpool.py", line 416, in _make_request
  File "http/client.py", line 1344, in getresponse
  File "http/client.py", line 306, in begin
  File "http/client.py", line 267, in _read_status
  File "socket.py", line 589, in readinto
  File "ssl.py", line 1071, in recv_into
  File "ssl.py", line 929, in read
KeyboardInterrupt
[6078] Failed to execute script garmin

Note that the Traceback doesn't show up until I cancel the process.

Additional context
Given the limitation of the API to download too much, would it make sense to add end_date's along with start_date's?

[tcgoetz]

Please attach a garmin.log file for a failing case.

[markbulk]

Just to be clear on what I did:

1. Updated config file with start_dates back to 01/01/2018
2. Reran sh download_create_dbs.sh without deleting anything that had already been downloaded.

It would appear that the third time is a charm (despite the first two failing at the same spot). 2.5 hours later and I have verified that everything is in order. Sorry for the bother.