[Tilesathome] Towards a more efficient lowzoom (with figures)

[Martijn van Oosterhout]

Hi,

I've done some more research into the idea of using latest.txt.gz and
here is what I've found so far. According to latest.tar.gz (not a
exactly an oracle but...):

1. 79 zoom-6 tiles are out of date, with anything from 1 to 500+ newer
zoom-12 tiles.
2. Total number of zoom-12 tiles newer than the corresponding zoom-6 tiles: 3914
3. Number of downloads needed to get enough info to generate to level 6: 14053
4. Tiles that would be updated on the server: 4661

Standard lowzoom downloads 4096 and generates 1365 tiles to upload, so
as you can see it's a significant win. There are however problems:

- The above figure is seriously underestimated because it doesn't
include changed blank tiles. Long term that's not a big deal, but
right now it is. To work that out we need data on the blank tiles DB.
- There is the race condition mentioned. This can be solved by being
more careful about how timestamps on files are managed (for example:
the DB store the timestamp of the file rather than when it was
uploaded (or both)).
- This is basically from a period where not much lowzoom is done. If
this gets going it will need less daily work to keep up to date.
- Most of the processing time in this script is reading the
latest.txt.gz. I would propose creating an additional file:
latext-lowzoom.txt.gz. It's much much smaller and can be processed
under a second.

This isn't going to replace lowzoom in the short-term, but long-term
it's going to make the process a lot less manual.

Have a nice day,
-- 
Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/