Hi guys...<br><br>I looked at California counties (for no particular reason). For many of the smaller ones, I saw really no difference in performance or memory with or without the patch (they run in under 10 seconds or so).. But ones that were 20M zipped, really expand exponentially, and in memory can take 1G or more. I'm not at my main computer (and I'll be away for another
1.5 weeks), but I remember Kern County was one where the patches that saved 100s of megabytes. (why kern county is huge I have no idea... it's mostly desert and oil wells!). So some files have a lot more complexity, why I'm not exactly sure -- I didn't look into it.
<br><br>once you qualify my patch, I'd like to look into "phase 2" in performance improvements -- optimizing the xml generation part. But let me know when you are ready.. I don't want to stomp on your work.
<br><br>thanks!<br><br>--nickg<br><br><br><br><br><div><span class="gmail_quote">On 9/16/07, <b class="gmail_sendername">Dave Hansen</b> <<a href="mailto:email@example.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
firstname.lastname@example.org</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On Sat, 2007-09-15 at 15:09 -0400, Nick Galbreath wrote:<br>> Hi everyone,<br>><br>> This is my first post to dev.<br>><br>> Recently I've been working on speeding up USA TIGER data conversion to<br>> OSM format. I was able to take Dave Hansen's code
<br>> (<a href="http://wiki.openstreetmap.org/index.php/TIGER" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://wiki.openstreetmap.org/index.php/TIGER</a>) and run it on my home<br>> Mac, with minimal issues. nice!
<br><br>Hi Nick,<br><br>Thanks for looking into this so deeply. Your version of
tiger.rb<br>definitely runs faster than what we have now, so that's a great<br>improvement.<br><br>But, I'm wondering where all of the memory savings that you're seeing<br>come from. I just tested one county for now (Lincoln, OR), and the old
<br>version of tiger.rb used ~95MB of memory. Your version used ~89MB.<br><br>What counties were you testing with, and what were the actual changes in<br>sizes that you were seeing?<br><br>I've also tweaked the ruby scripts as I've gone along so I wonder if
<br>what we have are slightly different. I know I've improved memory usage<br>over time.<br><br>-- Dave<br><br></blockquote></div><br>