[OSM-dev] TIGER upload automation

[Dave Hansen]

On Wed, 2007-08-29 at 23:35 +0100, Tom Hughes wrote:
> In message <1188412563.28903.111.camel at localhost>
>           Dave Hansen <dave at sr71.net> wrote:
> 
> > On Wed, 2007-08-29 at 19:17 +0100, Tom Hughes wrote:
> >
> > > Incidentally, I only said that dev wasn't the best place to do it
> > > not that we shouldn't try and run it on one of our servers. How
> > > much CPU and memory does the uploader use anyway?
> > 
> > Right now, uploading one medium-large county, it has an 18MB memory
> > footprint.  It uses ~2% of my cpu sporadically.  Not much cpu load at
> > all.
> 
> That sounds much better than I expected - I thought it would use more
> memory than that for the ID mapping. Is it doing that on disk now then?

The bulk_upload.pl script keeps a cache file, but it also keeps the data
in a perl hash.  The memory footprint could potentially grow, so I'll
keep an eye on it as my test upload gets farther along and the id
mapping hash gets bigger.

The process is up to 20MB, and the upload is ~10% done.  But, even if
the memory footprint went to 200MB, I bet that most of the memory is
pretty rarely accessed and will end up swapped out pretty fast.  I'll
keep an eye on it.

-- Dave