Use high quality USGS elevation data

[graue]

USGS makes Bay Area elevation data available as a few hundred MB dataset, with 1/9 arc angle (about 1 meter) resolution. Earth's surface, not trees/buildings. Would be better than SRTM (~30m resolution) that we currently use which has lots of artifacts, especially downtown. Not sure if GraphHopper can use this GeoTIFF format or how to convert it.

[rsarathy]

I just got Graphhopper to successfully ingest both the 1/3 arcsecond and 1/9 arcsecond USGS DEMs.

They use Float32 to encode elevation, so converting them to Int16 beforehand with gdal_translate is necessary. This is because the third-party TiffImageDecoder library that Graphhopper uses to parse elevation data only accepts < 2 byte datatypes, as it maps values to the short datatype in Java. This shouldn't be too much of an issue for us, at most, we're losing the fractional portion of a meter.

For 1/3 arcsecond datasets:

$  gdal_translate -ot Int16 ned13_raw_usgs.tif ned13.tif

For 1/9 arcsecond datasets (USGS only provides them in .img formats, we want a GeoTiff):

$ gdal_translate -of gtiff -ot Int16 ned19_raw_usgs.img ned19.tif

@Andykmcc, this does create an additional data dependency for us. 1/3 AS tiles come in 1x1 latitude-longitude, so to cover the 9-county region of the Bay Area, this will be ~3-4 different TIFFs IIRC. 1/9 AS tiles come in 1/4x1/4 lat-lon, so this would be closer to 50.

Regardless of the data that we decide to work with, I'll implement additional logic to retrieve elevation from the respective TIFFs.

[Andykmcc]

My initial thoughts are:

This data is static so there's no need to pull the source data in an ongoing basis. By the same toke, that means we can preprocess that data by hand. Basically, download the source data to your machine, write a (dirty) script to process it into whatever state it needs to be for Graphhopper to ingest it. Commit the script(s) so place so we can recycle them as needed. Then we can store the process elevation data in min.io along side the PBF and GTFS. We can add a third "downloader" here to fetch the processed elevation data. if, in the future there is a reason/desire to automate this process we can do so.

Questions:

• When do we need the elevation data? during graph build presumably, but do we need to to run graphhopper in server mode? my recollection is that these files are fairly large, so if we can avoid needing to downloading them every time GH starts that would be cool. if we need the data for both "import" and "server" it's not a deal breaker, it just means slower boot times for graphhopper.
• What happens if this data is missing? like if i standup BH in a city where the higher quality data isn't available or i just don't feel like going through this process. will it just fall back to SRTM or no data? mostly i just want to know will graphhoppper still work.

[rsarathy]

+1 on how to manage and preprocess the data. I don't anticipate the new data adding much more complexity to our systems. I'll write some basic scripts as part of my existing PR.

Data Precision

We will need to decide if we want to use 1/3 AS or 1/9 AS elevation data moving forward.

I am personally in favor of 1/3 AS because of its wider availability, and it does seem, at face value, to solve the issues that we were seeing with SRTM datasets. It will also have a smaller footprint. Our data just needs to be good enough!

Responses

1. We need the elevation data during the graph build step. When parsing OSM ways, Graphhopper will use the config-specified ElevationProvider (we use srtm, which is the default) to calculate each way's grade and elevation change. Some back-of-the-envelope math:

• The 1/3 AS 1x1 tiles are ~240M each. This would translate to ~1.5G of elevation artifacts.
• The 1/9 AS 0.25x0.25 tiles are ~135M each. This would translate to ~8G of elevation artifacts.

2. Graphhopper has an ElevationProvider implementation called MultiSourceElevationProvider. They addressed your concern in this implementation; basically they combined a preferred elevation provider which sources higher resolution elevation data where possible, and a fallback "global" provider that sources lower quality data with greater availability. We can use MultiSourceElevationProvider, passing our new ElevationProvider implementation and the existing SRTMElevationProvider as the fallback. IIRC BikeHopper will still run if it doesn't have elevation data for a given area, it will just return 0 for any getEle() call.

[rsarathy]

From the USGS website:
1/3 arc-second: This is the highest resolution seamless DEM dataset for the U.S. with full coverage of the 48 conterminous states, Alaska, Hawaii, and U.S. territories. Ground spacing is approximately 10 meters north/south, but variable east/west due to convergence of meridians with latitude.

1/9 arc-second: This dataset covers about 25 percent of the conterminous U.S. and is produced from 3-meter or higher resolution DEMs acquired by the USGS prior to January 2015. Horizontal coordinates are referenced to geographic coordinates (longitude, latitude). The 1/9 arc-second dataset will no longer be updated with newly acquired DEMs; however, it will continue to be distributed.

1-meter: This dataset was introduced in 2015 with limited coverage of the U.S., but will be expanding as new DEMs from 3DEP quality level 2 or better lidar data are acquired. Horizontal coordinates are referenced to the Universal Transverse Mercator projection.

[abhumbla]

I'm in favor of the 1/3 arcsecond data for now

[Andykmcc]

Sounds good. Thanks for the details. Im on the same page about 1/3.

[graue]

Am I right that the SRTM data we have is more or less accurate enough outside of downtown SF? Would it make sense to go with a hybrid to save download and disk?

[Andykmcc]

My understanding of rohit's work is that it is a hybrid between the higher quality data and srtm. 1.5gb is okay, it's about 6-8 seconds download time with the current setup, which is confined to the wired switch in your office. My design for the future cluster, v3, includes a local cache for these exact types of things. If it was an order of magnitude bigger I'd be more worried (e.g. the 1/9 arc).

[rsarathy]

We can limit the artifacts' footprint even more with the MultiSourceElevationProvider. My plan right now is to only use a single, 1/3 AS lat-lon graticule (37-38, -122-123), and use SRTM for anything outside of that area. This single tile's size is 234M.

[rsarathy]

Keeping this open until the new multi-source EP is in prod.

[rsarathy]

New router issues:

• JFK Promenade (uses footpath)
• Market & Laguna segment

[rsarathy]

New router issues:

• JFK Promenade (uses footpath)
• Market & Laguna segment

Promenade issue:

[rsarathy]

How to convert a NED 1/9 arcsecond .img file to a Graphhopper-compatible TIFF:

1. Use gdal_translate to convert the file type to a TIFF, and to change the datatype to a short:

$ gdal_translate \                                                     
  -co GTiff \     
  -ot Int16 \
ned19_n37x75_w122x25_[...].img ned19_n37x75_w122x25.tif

2. Compress the file:

$ tar -czvf ned19_n37x75_w122x25.tgz ned19_n37x75_w122x25.tif