Upparat stuck in an endless loop

[dShapo]

Device 8ffghxz1 had several AWS IoT Jobs active, but upparat was stuck in an endless loop, see the log...

Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,455] INFO in __init__: State changed from download to fetch_jobs.                                                                                           
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,526] INFO in select_job: Job execution in progress: 0-6-3-beta-tester                                                                                       
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,536] INFO in __init__: State changed from fetch_jobs to select_job.                                                                                         
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,637] INFO in verify_job: Redo job process                                                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,653] INFO in __init__: State changed from select_job to verify_job.                                                                                         
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,689] INFO in __init__: State changed from verify_job to download.                                                                                           
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,763] INFO in download: Hook successfully completed. Download now allowed.                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,783] INFO in download: Downloading job to /tmp/upparat/0-6-3-beta-tester.                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,952] WARNING in download: URL has expired. Starting over.                                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,990] INFO in __init__: State changed from download to fetch_jobs.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,056] INFO in select_job: Job execution in progress: 0-6-3-beta-tester                                                                                       
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,074] INFO in __init__: State changed from fetch_jobs to select_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,138] INFO in verify_job: Redo job process                                                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,155] INFO in __init__: State changed from select_job to verify_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,182] INFO in __init__: State changed from verify_job to download.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,276] INFO in download: Hook successfully completed. Download now allowed.                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,305] INFO in download: Downloading job to /tmp/upparat/0-6-3-beta-tester.                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,556] WARNING in download: URL has expired. Starting over.                                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,609] INFO in __init__: State changed from download to fetch_jobs.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,683] INFO in select_job: Job execution in progress: 0-6-3-beta-tester                                                                                       
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,694] INFO in __init__: State changed from fetch_jobs to select_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,775] INFO in verify_job: Redo job process                                                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,781] INFO in __init__: State changed from select_job to verify_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,810] INFO in __init__: State changed from verify_job to download.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,883] INFO in download: Hook successfully completed. Download now allowed.
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,923] INFO in download: Downloading job to /tmp/upparat/0-6-3-beta-tester.
Aug 25 12:32:23 8ffghxz1 sh[219]: [2020-08-25 12:32:23,054] WARNING in download: URL has expired. Starting over.

Canceling older Jobs in cloud enabled device to fetch and execute the latest one
Bundle version: caru-version-0.5.6-2020-04-16-142853-456dec37RELEASE-imx6ul-caru-v-1-2-0
Upparat version: 1.4 - git commit hash = a5beb33b42119692091cb9d80140284d91326e9f

[livioso]

Assumption: Corrupted job w/ an URL that points to a bundle on S3 which does not exist anymore, thus constantly seeing a HTTP 403 (=> WARNING in download: URL has expired).

There is no way for us to distinguish between URL expiration, missing or access issues (policies) thus what would be the better way is to maybe have some backoff here between retries.