[OSM-dev] TIGER data show OSM getting slower??

Dave Hansen dave at sr71.net
Sun Sep 16 21:55:34 BST 2007


I've been running my TIGER statistics generation program about every 10
minutes for the duration of the TIGER upload (16 days so far).  I've
been tracking things like the number of objects uploaded per second and
the expected time to complete the TIGER upload.

Note that I've been hand-generating these so they won't get updated
automatically.  In this graph, the expected time in days before
completing the TIGER upload is shown:

http://dev.openstreetmap.org/~daveh/tiger/graphs/days.png

There were some huge swings at first because I was still generating
TIGER data, and the stats were only showing the completion times for
generated TIGER data (which was growing).  I think that huge drop
occurred when I move the upload from my server over to dev.

Note, however, that the upload is slowly taking more and more time.  The
server is slowing down faster than we can upload data.  

This is particularly apparent when you look at the per-second rate of
uploads.  This is simply the number of API transactions that are
occurring per second:

http://dev.openstreetmap.org/~daveh/tiger/graphs/persec.png

It's pretty obvious that we have a downward trend here.  You can also
see this trend in the individual county uploads.  They used to go each
at ~10 objects/sec.  Right now, they're managing 4.5.

I think the same trends are showing up in the munin data:

http://munin.openstreetmap.org/openstreetmap/www.openstreetmap-lighttpd_processes.html

So, is TIGER+AND causing too much load to the OSM infrastructure right
now?  AND will be done soon, but TIGER certainly won't be.  Plus, we're
not even _converging_ on a time when the upload will finish.  Should I
slow the upload down a bit?

-- Dave
-- Dave





More information about the dev mailing list