[OSM-dev] TIGER upload automation

Robert (Jamie) Munro rjmunro at arjam.net
Thu Aug 30 18:13:37 BST 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Hansen wrote:
> On Thu, 2007-08-30 at 09:37 +0200, spaetz wrote:
>> On Wed, Aug 29, 2007 at 07:17:24PM +0100, Tom Hughes wrote:
>>>>> 28 years is quite a long time. Could you be given an account on
>>>>> dev.openstreetmap.org, and run your scripts there, where they will have
>>>>> the minimum network latency? Hopefully it will run a lot faster.
>>>> I already asked.  dev is evidently to overloaded for new work to be
>>>> placed on it at this time.
>>> Incidentally, I only said that dev wasn't the best place to do it
>>> not that we shouldn't try and run it on one of our servers. How
>>> much CPU and memory does the uploader use anyway?
>> Dave has a dev account now, so he can try if it goes any faster. If
>> his Uploads hurt t at h I'll do likewise to him :-)
> 
> So, the upload from dev is proceeding a wee bit faster: about 20 times!

So that's still about 500 days to do the whole thing, or about a year
and a half. It's worth putting a few counties in this way so that we
have some tiger data to look at and think about, and it's probably
practical to put in the AND data this way, but we really need to find
another order of magnitude improvement.

> As a result, it's chewing up more CPU than it was on my laptop (~25% of
> a cpu). But, the memory footprint is still staying pretty small.

If you run two uploads in parallel, how does that change things? It
could be anywhere between both uploads going half the speed or even
slower as they use up memory that could be better used by the other,
meaning no overall improvement, or it might run twice as fast overall,
which would bring the total wait down to 9 months, which is worth doing.

Can we see the dev server versions of these progress pages:
http://www.sr71.net/~dave/osm/tiger/status.html
http://www.sr71.net/~dave/osm/tiger/upload-queue.txt

Robert (Jamie) Munro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG1vq+z+aYVHdncI0RAqkZAKCCqniO1XjgXDyskgZUcO6w3U9eOgCeP2j0
f8yIJyGWVzpurdxWBzTrR7M=
=Vk9x
-----END PGP SIGNATURE-----




More information about the dev mailing list