Skip re-download of DEM and mask if already found

[rzinke]

Skip re-downloading a DEM if --demfile=download and glo_90_uncropped.vrt exists; and skip downloading and formatting a water mask if the <mask_name>.msk file exists.

[pep8speaks]

Hello @rzinke! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-10-04 00:49:03 UTC

[alexfore]

I've tested this and it seems to work as expected for me. I also verified that the regression tests all pass with this branch.

[sssangha]

I see a couple of issues that need to be patched @rzinke .

When I download a DEM from scratch, the temporary directory containing the tiles glo_90_tiles is not deleted.

Also, when running a command with an existing command, I get the following error:

/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/ARIA-tools_dev/tools/ARIAtools/__init__.py:9: RuntimeWarning: ARIA-tools is not installed! Install in editable/develop mode via (from the top of this repo): python -m pip install -e .
  warnings.warn(
2024-10-08 09:27:28,502 - ariaExtract.py - INFO - Extract Product Function
2024-10-08 09:27:28,909 - ARIAtools.product - INFO - All (3) GUNW products meet spatial bbox criteria.
2024-10-08 09:27:28,909 - ARIAtools.product - INFO - Group GUNW products into spatiotemporally continuous interferograms.
2024-10-08 09:27:28,932 - ARIAtools.product - INFO - All (3) interferograms are spatially continuous.
2024-10-08 09:27:28,932 - ariaExtract.py - INFO - Thread count specified for gdal multiprocessing = 2
2024-10-08 09:27:28,932 - ariaExtract.py - INFO - Extracting and merging product bounding boxes
2024-10-08 09:28:04,152 - ariaExtract.py - INFO - Download/cropping mask
2024-10-08 09:28:04,153 - ARIAtools.util.mask - DEBUG - prep_mask
2024-10-08 09:28:08,340 - ARIAtools.util.mask - DEBUG - Using user specified mask test_mask_DEM/mask/esa_world_cover_2021.msk
2024-10-08 09:28:08,340 - ARIAtools.util.mask - DEBUG - The mask you specified already exists in /u/duvel-d0/ssangha/ARIA_tools_tests/D071_test/test_mask_DEM/mask, using the existing one...
Traceback (most recent call last):
  File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/ARIA-tools_dev/tools/bin/ariaExtract.py", line 316, in <module>
    main()
  File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/ARIA-tools_dev/tools/bin/ariaExtract.py", line 249, in main
    maskfilename = ARIAtools.util.mask.prep_mask(**mask_dict)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/ARIA-tools_dev/tools/ARIAtools/util/mask.py", line 157, in prep_mask
    shutil.rmtree(temp_workdir)
  File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/envs/ARIA-tools_dev/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/envs/ARIA-tools_dev/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/u/trappist-r0/ssangha/conda_installation/miniforge/miniforge/envs/ARIA-tools_dev/lib/python3.12/shutil.py", line 662, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=dirfd)
OSError: [Errno 39] Directory not empty: '/u/duvel-d0/ssangha/ARIA_tools_tests/D071_test/test_mask_DEM/mask/tmp_dir'

To replicate this, you can run:

ariaDownload.py --bbox "34.6 34.8 -118.1 -117.9" --track 71 --output Download --start 20211214 --end 20220102
ariaExtract.py -f "products/*.nc" -l 'amplitude,unwrappedPhase' -w test_mask_DEM -d Download -m Download -of ISCE --log-level 'debug'
ariaExtract.py -f "products/*.nc" -l 'coherence' -w test_mask_DEM -d test_mask_DEM/DEM/glo_90.dem -m test_mask_DEM/mask/esa_world_cover_2021.msk -of ISCE --log-level 'debug'

I'll try to look at this on my end, but my bandwidth is limited as I'll be out of the office Thurs-Friday this week.

Also tagging in @ehavazli