[Tilesathome] Active queue

Dirk-Lüder Kreie osm-list at deelkar.net
Tue Jul 3 14:44:01 BST 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthias Julius schrieb:
> Hi all,
> 
> I was just wondering about the high numbers in the active (1.3k) and
> new (1.36k) queues.  With the current rate of processing uploads of
> 135 uploads/hour in six hours the server will be able to process 810
> uploads.  The rest (490) will expire before the server will have a
> chance to look at them and they will be put in the new queue.  They
> will be uploaded and processed anyway.  I believe they will be cleared
> from the queue after that, but if another client has picked up the job
> from the new queue in the meantime it will be rendered and uploaded
> twice.
>
> If my assumptions are correct this is not very effective.  I think
> expired jobs should really be discarded, possibly bey the client.
> Also, the active queue should be limited.  There is no point in
> handing out more jobs to the clients than the server can process.  600
> might be a good number.
> 
> Any opinions?


The "active" queue is currently inflated by the fact that the API did
not respond for about an hour today and thus all the jobs failed. Since
there is currently no mechanism for the client to tell the server the
job has failed the active queue has to requeue the job somehow. The
average for the active queue is ~580 jobs according to munin which is
well within the limits of what can be processed.

Another thing is that if a job takes just over 6 hours to be uploaded it
doesn't get counted *at all* even when it's inserted into the tileset. I
think the "done" field should also encompass the jobs not found in
"active" and perhaps be able to remove jobs from the "new" queue
(recently timed out "active" jobs) but not the "pending" queue (jobs
requested through the request API)

- --

Dirk-Lüder "Deelkar" Kreie
Bremen - 53.0952°N 8.8652°E

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGilKhFUbODdpRVDwRAmkGAKDOSdmxzDXawr6wV4QR1Kv+AO80BQCgmI+f
Yz7F4lrJ0BVutprhUozj41g=
=C6ul
-----END PGP SIGNATURE-----




More information about the Tilesathome mailing list