<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Definitely worth checking out. When I setup the system I didn't install any of the CentOS Postgres packages and instead relied on the pgdg84 available ones.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">#> ldd /usr/local/bin/osm2pgsql | grep libpq<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> libpq.so.5 => /usr/lib64/libpq.so.5 (0x00002aab6b07c000)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">-------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Now for the embarrassing part. While the machine I'm running on has 16GB of RAM it lives in a virtual machine and it received a RAM cap of 372MB. Whoops. After finding the right logs to check (/var/log/pgsql is a red herring and I have a non-standard data directory location) all the problems were being caused by *massive* swapping and eventual shared memory allocation errors which stacked up until postgres just gave up the ghost and couldn't auto recover any more.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I asked our admin to bump me up to 8GB and the Austria import completed beautifully in only 6 minutes without even a hint of an error. As a comparison, it took 45 minutes before for Austria to fail. Postgres logs are clean as well.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I have started the planet import with --slim turned back on and will write back with how it works. My apologies for having missed this one. Also, you guys are lightning quick on the draw with really helpful questions and replies. Your help has been much appreciated!</div></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">— Seth</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br class="khtml-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">P.S. In the time it took to write up this email and verify I haven't missed anything else, I've pulled in 40000k points and eaten up 4GB of RAM. It's running quite a bit faster than I had expected.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span class="Apple-style-span" style="font-size: medium; "><font class="Apple-style-span" color="#808080" face="Verdana" size="2"><span class="Apple-style-span" style="font-size: 10px;"><b><br></b></span></font></span></div></span></div></span></div></span></div></span></span></div><div><div>On Apr 15, 2010, at 2:49 PM, Jon Burgess wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On Thu, 2010-04-15 at 14:24 -0700, Seth Voltz wrote:<br><blockquote type="cite">Jon and Frederik: good hunch, the Austrian import failed with:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--------------------------------<br></blockquote><blockquote type="cite">#> osm2pgsql --database world --username gis --password --verbose<br></blockquote><blockquote type="cite">austria.osm.bz2<br></blockquote><blockquote type="cite">... (cut) ...<br></blockquote><blockquote type="cite">Reading in file: austria.osm.bz2<br></blockquote><blockquote type="cite">Processing: Node(6760k) Way(0k) Relation(0k)planet_osm_point - bad<br></blockquote><blockquote type="cite">result during COPY, data 430058975 \N \N \N \N<br></blockquote><blockquote type="cite"> \N \N \N \N \N \N \N \N \N<br></blockquote><blockquote type="cite"> \<br></blockquote><blockquote type="cite">N \N \N \N \N \N \N \N<br></blockquote><blockquote type="cite"> bus_stop \N \N \N \N \N \N \N<br></blockquote><blockquote type="cite"> \N \N \N \N Altöttinger Straße/Aldi \N \N<br></blockquote><blockquote type="cite"> \<br></blockquote><blockquote type="cite">N \N \N \N \N \N \N \N \N<br></blockquote><blockquote type="cite"> \N \N \N \N \N \N \N \N \N<br></blockquote><blockquote type="cite"> \N SRID=900913;POINT(1398018.41536173 6112460.23312236)<br></blockquote><blockquote type="cite">... (cut) ...<br></blockquote><blockquote type="cite">--------------------------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I double checked the database: everything, including my GIS templates<br></blockquote><blockquote type="cite">and the working database are UTF8:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--------------------------------<br></blockquote><blockquote type="cite">#> psql -U gis -l<br></blockquote><blockquote type="cite">... (cut) ...<br></blockquote><blockquote type="cite">world | gis | UTF8 | en_US.UTF-8 | en_US.UTF-8 |<br></blockquote><blockquote type="cite">--------------------------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Also, I just noticed that when it fails it also crashes the whole<br></blockquote><blockquote type="cite">PostgreSQL server requiring a `service postgresql restart` to get<br></blockquote><blockquote type="cite">going again. This started when I began using the --verbose switch.<br></blockquote><blockquote type="cite">Failed imports with and without --slim and without --verbose just fail<br></blockquote><blockquote type="cite">but leave the database alone.<br></blockquote><br>Anything which crashes the database backend sounds more like an issue in<br>Postgresql rather than osm2pgsql.<br><br>One thing you might want to try is to recompile osm2pgsql against the<br>PostgreSQL-8.4 client library. There is a good chance that a binary<br>built in the normal way will be linking against the standard system<br>library (probably /usr/lib/libpq.so.4) not the postgresql-8.4 (probably<br>libpq.5 somewhere). <br><br>To find out the library it is using try:<br>$ ldd ~/bin/osm2pgsql | grep libpq<br> libpq.so.5 => /usr/lib64/libpq.so.5 (0x00007f6311499000)<br><br>The above output is from a F-12 system using PostgreSQL-8.4.3<br><br>You will need to install the pgdg postgresql-devel package and maybe<br>tweak the include path in the Makefile to pick up the right version when<br>you recompile.<br><br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">The log file in /var/log/pgsql is empty (0 bytes).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Matt & Peter: good catch, it was a typo. I have 8.4.3 from the pgdg84<br></blockquote><blockquote type="cite">repository which I setup with:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--------------------------------<br></blockquote><blockquote type="cite">#> rpm -ivh<br></blockquote><blockquote type="cite"><a href="http://yum.pgsqlrpms.org/reporpms/8.4/pgdg-centos-8.4-2.noarch.rpm">http://yum.pgsqlrpms.org/reporpms/8.4/pgdg-centos-8.4-2.noarch.rpm</a><br></blockquote><blockquote type="cite">--------------------------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">— Seth<br></blockquote><blockquote type="cite"><br></blockquote></div></blockquote></div><br></body></html>