[OSM-dev] improving TIGER-to-OSM memory usage/performance

Nick Galbreath nickg at modp.com
Mon Sep 17 16:32:51 BST 2007


Hi guys...

I looked at California counties (for no particular reason).  For many of the
smaller ones, I saw really no difference in performance or memory with or
without the patch (they run in under 10 seconds or so)..  But ones that were
20M zipped, really expand exponentially, and in memory can take 1G or more.
  I'm not at my main computer (and I'll be away for another 1.5 weeks), but
I remember Kern County was one where the patches that saved 100s of
megabytes.  (why kern county is huge I have no idea... it's mostly desert
and oil wells!).  So some files have a lot more complexity, why I'm not
exactly sure -- I didn't look into it.

once you qualify my patch, I'd like to look into "phase 2" in performance
improvements -- optimizing the  xml generation part.  But let me know when
you are ready.. I don't want to stomp on your work.

thanks!

--nickg




On 9/16/07, Dave Hansen <dave at sr71.net> wrote:
>
> On Sat, 2007-09-15 at 15:09 -0400, Nick Galbreath wrote:
> > Hi everyone,
> >
> > This is my first post to dev.
> >
> > Recently I've been working on speeding up USA TIGER data conversion to
> > OSM format.  I was able to take Dave Hansen's code
> > (http://wiki.openstreetmap.org/index.php/TIGER) and run it on my home
> > Mac, with minimal issues.  nice!
>
> Hi Nick,
>
> Thanks for looking into this so deeply.  Your version of tiger.rb
> definitely runs faster than what we have now, so that's a great
> improvement.
>
> But, I'm wondering where all of the memory savings that you're seeing
> come from.  I just tested one county for now (Lincoln, OR), and the old
> version of tiger.rb used ~95MB of memory.  Your version used ~89MB.
>
> What counties were you testing with, and what were the actual changes in
> sizes that you were seeing?
>
> I've also tweaked the ruby scripts as I've gone along so I wonder if
> what we have are slightly different.  I know I've improved memory usage
> over time.
>
> -- Dave
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20070917/5fb0e45f/attachment.html>


More information about the dev mailing list