<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div><div><div>On 25 Apr 2010, at 11:25, Jon Burgess wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On Sun, 2010-04-25 at 10:51 -0700, Simon Gornall wrote:<br><blockquote type="cite">So, I'd set everything up on my Ubuntu linux box, and decided to time<br></blockquote><blockquote type="cite">the import for posterity...<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">www:/opt/gis> time osm2pgsql -s -v -U simon -d gis -C 7000<br></blockquote><blockquote type="cite">data/planet-latest.osm <br></blockquote><blockquote type="cite"> osm2pgsql SVN version 0.66-<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Using projection SRS 900913 (Spherical Mercator)<br></blockquote><blockquote type="cite"> Setting up table: planet_osm_point<br></blockquote><blockquote type="cite"> Setting up table: planet_osm_line<br></blockquote><blockquote type="cite"> Setting up table: planet_osm_polygon<br></blockquote><blockquote type="cite"> Setting up table: planet_osm_roads<br></blockquote><blockquote type="cite"> Mid: pgsql, scale=100, cache=7000MB, maxblocks=896001*8192<br></blockquote><blockquote type="cite"> Setting up table: planet_osm_nodes<br></blockquote><blockquote type="cite"> NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index<br></blockquote><blockquote type="cite"> "planet_osm_nodes_pkey" for table "planet_osm_nodes"<br></blockquote><blockquote type="cite"> Setting up table: planet_osm_ways<br></blockquote><blockquote type="cite"> NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index<br></blockquote><blockquote type="cite"> "planet_osm_ways_pkey" for table "planet_osm_ways"<br></blockquote><blockquote type="cite"> Setting up table: planet_osm_rels<br></blockquote><blockquote type="cite"> NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index<br></blockquote><blockquote type="cite"> "planet_osm_rels_pkey" for table "planet_osm_rels"<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Reading in file: data/planet-latest.osm<br></blockquote><blockquote type="cite"> Processing: Node(593072k) Way(45376k) Relation(484k)<br></blockquote><blockquote type="cite"> Node stats: total(593072533), max(696096737)<br></blockquote><blockquote type="cite"> Way stats: total(45376969), max(55410575)<br></blockquote><blockquote type="cite"> Relation stats: total(484528), max(555276)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Going over pending ways<br></blockquote><blockquote type="cite"> processing way (13679k)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Going over pending relations<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> node cache: stored: 593072533(100.00%), storage efficiency:<br></blockquote><blockquote type="cite"> 86.21%, hit rate: 100.00%<br></blockquote><blockquote type="cite"> Committing transaction for planet_osm_line<br></blockquote><blockquote type="cite"> Committing transaction for planet_osm_polygon<br></blockquote><blockquote type="cite"> Committing transaction for planet_osm_roads<br></blockquote><blockquote type="cite"> Sorting data and creating indexes for planet_osm_roads<br></blockquote><blockquote type="cite"> Sorting data and creating indexes for planet_osm_polygon<br></blockquote><blockquote type="cite"> Sorting data and creating indexes for planet_osm_line<br></blockquote><blockquote type="cite"> Committing transaction for planet_osm_point<br></blockquote><blockquote type="cite"> Sorting data and creating indexes for planet_osm_point<br></blockquote><blockquote type="cite"> ANALYZE planet_osm_roads;<br></blockquote><blockquote type="cite"> failed: ERROR: could not access status of transaction<br></blockquote><blockquote type="cite"> 4026772988<br></blockquote><blockquote type="cite"> DETAIL: Could not open file "pg_clog/0F00": No such file or<br></blockquote><blockquote type="cite"> directory.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> Error occurred, cleaning up<br></blockquote><blockquote type="cite"> 12523.240u 403.910s 7:13:16.49 49.7% 0+0k 266220448+0io<br></blockquote><blockquote type="cite"> 10515pf+0w<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I'm a bit concerned about the red bit above...<br></blockquote><br>The error is coming from the postgresql server, you should probably<br>check its logs (/var/log/postgresql/postgresql-8.3-main.log). Perhaps<br>you ran out of disk space or exceeded some resource limit?<br></div></blockquote></div><br><div>Well, the logs don't actually tell me much more than the error did:</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 40px; border-top-style: none; border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; ">2010-04-24 20:19:22 PDT LOG: database system is ready to accept connections<br>2010-04-24 20:19:22 PDT LOG: incomplete startup packet<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_point" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_point;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_point_tmp" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_point_tmp;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_line" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_line;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_line_tmp" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_line_tmp;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_polygon" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_polygon;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_polygon_tmp" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_polygon_tmp;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_roads" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_roads;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_roads_tmp" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_roads_tmp;<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_nodes" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_nodes<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_ways" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_ways<br>2010-04-24 20:22:25 PDT ERROR: table "planet_osm_rels" does not exist<br>2010-04-24 20:22:25 PDT STATEMENT: DROP TABLE planet_osm_rels<br>2010-04-25 03:35:40 PDT ERROR: could not access status of transaction 4026772988<br>2010-04-25 03:35:40 PDT DETAIL: Could not open file "pg_clog/0F00": No such file or directory.<br>2010-04-25 03:35:40 PDT STATEMENT: ANALYZE planet_osm_roads;<br><span class="Apple-tab-span" style="white-space: pre; "> </span><br>2010-04-25 03:35:42 PDT LOG: unexpected EOF on client connection<br>2010-04-25 03:35:42 PDT LOG: unexpected EOF on client connection<br>2010-04-25 03:35:42 PDT LOG: unexpected EOF on client connection<br>2010-04-25 03:36:37 PDT LOG: could not send data to client: Broken pipe<br>2010-04-25 03:37:56 PDT LOG: could not send data to client: Broken pipe<br>2010-04-25 03:38:11 PDT LOG: could not send data to client: Broken pipe</blockquote><div><div><br></div><div>... but this is the first time I've set up postgresql. Perhaps I'll just wipe everything and start again. I'm pretty sure it didn't run out of disk space (there's 6TB free on /opt :) and it didn't run out of memory either:</div><div><br></div><div>[graph removed to fit within 40k limit on mail-size]. Guess you'll have to trust me :)</div><div><br></div><div>... you can see there's alway some ram still being used for 'cached'... The swap does start to get used, but there's 48G configured, and we're nowhere near that.</div><div><br></div><div>Simon.</div><div><br></div></div></div></body></html>