[Tilesathome] TAH tile status checking

[Patrick Kilian]

Hi,

> Patrick (petschge) is also running some kind of checker which checks
> for outdated tiles (don't know exactly, what it is doing).
I had to check the source code to see which features are actually
implemented. *g*


> Since it makes no sense to run all these (and maybe more in the
> future) I would like to suggest to carry all ideas together and build
> a really good checker which runs on a server continuously (mine is
> only running when I'm at home).
> 
> Any opinions?
Makes sense. I uploaded the source code of my checker to
http://www.petschge.de/osm/oldtiles.sh.txt and I'd be willing to host
the new checker unless it has massive dependencies which I can't satisfy
with my debian old-stable box.

The basic algorithm (besides all the logging and stuff) is the following:

A.) Select uniform random x/y tile numbers and do checks:
1.) Check the png2tileinfo DB from T at h if we are out in the middle of
    the ocean. If we are skip that tile.
2.) Ask the server for the metadata of the tile. If the server serves
    this tile from the oceanfiles.dat (because it is empty land) don't
    check any further.
3.) If the tile is newer then 6 weeks it's new enough for now.
4.) Request the tile to be rerendered

B.) If a tile has to be rerendered the 8 neighbours are checked as well.

C.) If at least have of the neighbours have to be rerended check the 16
    next-to-neighbours of the tile too

D.) If it is more then 10 minutes that we checked the load level of the
    T at h server, then check the lenght of the rendering queue and sleep
    if it is greater then 10000. Also sleep an hour if the server fails
    to respond to the load query (it sometime does due to RoR
    strangeness)

E.) Sleep at least a second and start over with step A.


Check #1 might fail for tiles containing new islands but reduces the
load on the server by almost 2/3.
Check #2 is not strictly necessary but is a lot older then check #1 and
apparently does no harm.
Check #3 is obviously available for fine tuning, but we basically need
12 to 15 weeks to rerender every (non-empty) tile so half of that seems
to be a reasonable number.

Steps B and C increase the probability to find stale tiles a lot plus
the necessary metadata are cache hot on the server.

Step D was introduced to keep the queue full enough to keep all clients
busy despite the fluctuations from the regular tileexpiry code. On the
other hand it tries to keep the queue short enough that request from
other sources egt processed in a timely manner.



HTH,
Patrick "Petschge" Kilian