Expire using metatiles zoom levels?

[joto]

When a tile is expired on, say, zoom level 19, it is marked in the expire list in that zoom level 19. But really we are using metatiles, so always the whole metatile is getting expired. If there are a lot of tiles in the same metatile, they are all in the expire list. In the worst case there are 8x8=64 times the number of entries in the expire list than we would really need. We could just write out expire list on zoom level 16 instead.

So I propose some kind of option to render_expires (and maybe render_list) that says: interpret zoom levels as meta tile zoom levels, not tile zoom levels.

See also: osm2pgsql-dev/osm2pgsql#2190

[hummeltech]

Thank you for creating the issue @joto, I have begun to look into implementing this into render_expired and have come up with some ideas for preventing duplicate work based on your comment and reading the linked comments that I am testing out.

I'm not sure what you mean here though:

We could just write out expire list on zoom level 16 instead.

as metatiles do not contain tiles spanning multiple zoom levels, Could you please explain what you mean there?

[joto]

What I mean is that instead of writing out, say z19,x453726,y39265, I write out z16,x56715,y4908 in the expire list. z16,x56715,y4908 is the metatile that contains z19,x453726,y39265 (and many neighboring tiles). Then render_expired expires the metatile which contains z19,x453726,y39265. Basically we have to remove the code in render_expired which calculates the matatile from the tile it is given and instead we give it the metatile directly.

[hummeltech]

It seems like that is basically what is already being done, more or less, here's an example command:

printf "16/56715/4908\n17/113420/9816\n18/226860/19632\n19/453726/39265\n" | \
  render_expired --min-zoom 3 --max-zoom 19 --verbose --touch-from 0

1. The first line is read (16/56715/4908) and a loop is started on it for zoom levels 3 to 19 (min-zoom to max-zoom):

   ** Message: Read valid line: 16/56715/4908
   ** Message: Starting loop on 19/453720/39264 for zoom levels 3 to 19

   1. Zoom levels 19-6 are requested, and the metatiles do not exist on my disk, so they are skipped (processing would indeed occur if they were to exist)

      ** Message: Processing: 19/453720/39264
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/19/96/233/201/86/128.meta' (metatile does not exist)
      ** Message: Processing: 18/226860/19632
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/18/48/116/108/43/128.meta' (metatile does not exist)
      ** Message: Processing: 17/113430/9816
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/17/16/178/182/21/8.meta' (metatile does not exist)
      ** Message: Processing: 16/56715/4908
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/16/0/209/211/130/136.meta' (metatile does not exist)
      ** Message: Processing: 15/28357/2454
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/15/0/96/233/201/0.meta' (metatile does not exist)
      ** Message: Processing: 14/14178/1227
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/14/0/48/116/108/8.meta' (metatile does not exist)
      ** Message: Processing: 13/7089/613
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/13/0/16/178/182/0.meta' (metatile does not exist)
      ** Message: Processing: 12/3544/306
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/12/0/0/209/211/128.meta' (metatile does not exist)
      ** Message: Processing: 11/1772/153
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/11/0/0/96/233/136.meta' (metatile does not exist)
      ** Message: Processing: 10/886/76
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/10/0/0/48/116/8.meta' (metatile does not exist)
      ** Message: Processing: 9/443/38
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/9/0/0/16/178/128.meta' (metatile does not exist)
      ** Message: Processing: 8/221/19
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/8/0/0/0/209/128.meta' (metatile does not exist)
      ** Message: Processing: 7/110/9
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/7/0/0/0/96/136.meta' (metatile does not exist)
      ** Message: Processing: 6/55/4
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/6/0/0/0/48/0.meta' (metatile does not exist)

   2. Zoom levels 5-3 are requested, and the metatiles do exist on my disk, so they are touched (as per --touch-from 0)

      ** Message: Processing: 5/27/2
      ** Message: Touching 'file:///var/cache/renderd/tiles/default/5/0/0/0/16/128.meta'
      ** Message: Processing: 4/13/1
      ** Message: Touching 'file:///var/cache/renderd/tiles/default/4/0/0/0/0/128.meta'
      ** Message: Processing: 3/6/0
      ** Message: Touching 'file:///var/cache/renderd/tiles/default/3/0/0/0/0/0.meta'

2. The second line is read (17/113420/9816) and a loop is again started on it for zoom levels 3 to 19 (min-zoom to max-zoom):

   ** Message: Read valid line: 17/113420/9816
   ** Message: Starting loop on 19/453680/39264 for zoom levels 3 to 19

   1. Zoom levels 19-16 are requested, and the metatiles do not exist on my disk, so they are skipped (processing would indeed occur if they were to exist)

      ** Message: Processing: 19/453680/39264
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/19/96/233/201/54/0.meta' (metatile does not exist)
      ** Message: Processing: 18/226840/19632
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/18/48/116/108/27/128.meta' (metatile does not exist)
      ** Message: Processing: 17/113420/9816
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/17/16/178/182/5/136.meta' (metatile does not exist)
      ** Message: Processing: 16/56710/4908
      ** Message: Skipping 'file:///var/cache/renderd/tiles/default/16/0/209/211/130/8.meta' (metatile does not exist)

   2. Then, nothing needs to be done as the metatile for zoom level 15 and all below it have already been requested

      ** Message: Already requested metatile containing '15/28355/2454', moving on to next input line

3. The third line is read (18/226860/19632) and nothing needs to be done as its metatile and all above and below it have already been requested:

   ** Message: Read valid line: 18/226860/19632
   ** Message: Starting loop on 19/453720/39264 for zoom levels 3 to 19
   ** Message: Already requested metatile containing '19/453720/39264', moving on to next input line

4. Then the fourth line is read (19/453726/39265) and again nothing needs to be done as its metatile and all below it have already been requested:

   ** Message: Read valid line: 19/453726/39265
   ** Message: Starting loop on 19/453726/39265 for zoom levels 3 to 19
   ** Message: Already requested metatile containing '19/453726/39265', moving on to next input line