[OSM-talk] Is this normal ?

[Simon Gornall]

On 25 Apr 2010, at 11:25, Jon Burgess wrote:

> On Sun, 2010-04-25 at 10:51 -0700, Simon Gornall wrote:
>> So, I'd set everything up on my Ubuntu linux box, and decided to time
>> the import for posterity...
>> 
>> 
>> www:/opt/gis> time osm2pgsql -s -v -U simon -d gis -C 7000
>> data/planet-latest.osm  
>>        osm2pgsql SVN version 0.66-
>> 
>>        Using projection SRS 900913 (Spherical Mercator)
>>        Setting up table: planet_osm_point
>>        Setting up table: planet_osm_line
>>        Setting up table: planet_osm_polygon
>>        Setting up table: planet_osm_roads
>>        Mid: pgsql, scale=100, cache=7000MB, maxblocks=896001*8192
>>        Setting up table: planet_osm_nodes
>>        NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index
>>        "planet_osm_nodes_pkey" for table "planet_osm_nodes"
>>        Setting up table: planet_osm_ways
>>        NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index
>>        "planet_osm_ways_pkey" for table "planet_osm_ways"
>>        Setting up table: planet_osm_rels
>>        NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index
>>        "planet_osm_rels_pkey" for table "planet_osm_rels"
>> 
>>        Reading in file: data/planet-latest.osm
>>        Processing: Node(593072k) Way(45376k) Relation(484k)
>>        Node stats: total(593072533), max(696096737)
>>        Way stats: total(45376969), max(55410575)
>>        Relation stats: total(484528), max(555276)
>> 
>>        Going over pending ways
>>        processing way (13679k)
>> 
>>        Going over pending relations
>> 
>>        node cache: stored: 593072533(100.00%), storage efficiency:
>>        86.21%, hit rate: 100.00%
>>        Committing transaction for planet_osm_line
>>        Committing transaction for planet_osm_polygon
>>        Committing transaction for planet_osm_roads
>>        Sorting data and creating indexes for planet_osm_roads
>>        Sorting data and creating indexes for planet_osm_polygon
>>        Sorting data and creating indexes for planet_osm_line
>>        Committing transaction for planet_osm_point
>>        Sorting data and creating indexes for planet_osm_point
>>        ANALYZE planet_osm_roads;
>>         failed: ERROR:  could not access status of transaction
>>        4026772988
>>        DETAIL:  Could not open file "pg_clog/0F00": No such file or
>>        directory.
>> 
>>        Error occurred, cleaning up
>>        12523.240u 403.910s 7:13:16.49 49.7% 0+0k 266220448+0io
>>        10515pf+0w
>> 
>> 
>> 
>> 
>> I'm a bit concerned about the red bit above...
> 
> The error is coming from the postgresql server, you should probably
> check its logs (/var/log/postgresql/postgresql-8.3-main.log). Perhaps
> you ran out of disk space or exceeded some resource limit?

Well, the logs don't actually tell me much more than the error did:

2010-04-24 20:19:22 PDT LOG:  database system is ready to accept connections
2010-04-24 20:19:22 PDT LOG:  incomplete startup packet
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_point" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_point;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_point_tmp" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_point_tmp;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_line" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_line;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_line_tmp" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_line_tmp;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_polygon" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_polygon;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_polygon_tmp" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_polygon_tmp;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_roads" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_roads;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_roads_tmp" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_roads_tmp;
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_nodes" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_nodes
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_ways" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_ways
2010-04-24 20:22:25 PDT ERROR:  table "planet_osm_rels" does not exist
2010-04-24 20:22:25 PDT STATEMENT:  DROP TABLE planet_osm_rels
2010-04-25 03:35:40 PDT ERROR:  could not access status of transaction 4026772988
2010-04-25 03:35:40 PDT DETAIL:  Could not open file "pg_clog/0F00": No such file or directory.
2010-04-25 03:35:40 PDT STATEMENT:  ANALYZE planet_osm_roads;
	
2010-04-25 03:35:42 PDT LOG:  unexpected EOF on client connection
2010-04-25 03:35:42 PDT LOG:  unexpected EOF on client connection
2010-04-25 03:35:42 PDT LOG:  unexpected EOF on client connection
2010-04-25 03:36:37 PDT LOG:  could not send data to client: Broken pipe
2010-04-25 03:37:56 PDT LOG:  could not send data to client: Broken pipe
2010-04-25 03:38:11 PDT LOG:  could not send data to client: Broken pipe

... but this is the first time I've set up postgresql. Perhaps I'll just wipe everything and start again. I'm pretty sure it didn't run out of disk space (there's 6TB free on /opt :) and it didn't run out of memory either:

[graph removed to fit within 40k limit on mail-size]. Guess you'll have to trust me :)

... you can see there's alway some ram still being used for 'cached'... The swap does start to get used, but there's 48G configured, and we're nowhere near that.

Simon.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20100425/8f02728b/attachment.html>