[Tilesathome] Questions on tile updating process

[Andre Hinrichs]

On Sonntag 19 April 2009, Maarten Deen wrote:
> Ok, you're checking tiles as you go. My idea was to select the lowest
> renderdate and rerender that tile. That is less dirty but requires a
> database with the renderdates, or 16 million http requests to get all dates
> (which would make it an awful hack).

Well, I do not have direct database access and there is no lighweight API for 
requesting the data so I do have to request the status via http. But my idea 
is to do this a bit more intelligent and store already made quiries and 
requests in a local database with some additional info (quiry_date, tile_date, 
request_date, tile_changed_date) where the tile_changed_date comes from the 
OSM API. In case I trust the mechanism that tiles are automatically rerendered 
when the data in the tile has changed I will not have to check the status 
again until even this date is too old.

I was also thinking of storing the status of all tiles into the local 
database. This would initially require a complete quiry of all 16 million 
tiles. But afterwards it would only quiry empty tiles, if they are next to a 
tile with content. This would give me the possibility to leave empty sea and 
land tiles alone unless a neighbour tile gets content.

> > Requests are being sent with priority 4 and source
> > 'tile-reliability-checker'. Max age is currently set to one month since
> > the tiles seem to have been restored from a backup of 2009-03-04. At
> > least I cannot see tile dates older that date and younger than
> > 2009-04-14.
>
> There are. See tile (12, 2100, 1359) from 2009-01-14. There are more tiles
> in the Netherlands from januari.

Sorry, my logic was wrong. I meant younger than 2009-03-04 and older than 
2009-04-14. I haven't seen any tile date inbetween these dates.
Oldest tile seen recently was 2008-09-18.

> > This situation results in rather high rate of render requests. I think in
> > the future the situation will get better.
> >
> > The script takes approx. 0.4 seconds for each tile. Means, that for a
> > complete world check with 16 mio tiles it would take approx. 77 days to
> > finish. Isn't this a good result if the max age for a tile is set to two
> > months???
>
> Yup. Sounds as a reasonable hack for not having a direct access to all
> tile's renderdates.

Yup.

I'm currently implementing a check for the queue length. The queue length 
would be checked after each thousand requests and would go into a sleep loop 
(15 minutes sleep) if queue has more than 10,000 entries und would go out of 
sleep loop if queue length has fallen below 5,000 entries.


Regards,
Andre