[Imports] import crashed at "going over pending ways"

[Rowley, Shane K]

I just realized I forgot to include the import command I used. The server I used has 6 cores and 24GB RAM, and I configured 24GB swap as well.

$ osm2pgsql -d planet_osm -C 16000 -s -k -K --cache-strategy dense --number-processes 6 -r pbf planet-121205.osm.pbf

I also used the following config tweaks for performance..
shared_buffers = 6GB
synchronous_commit = off
checkpoint_segments = 64
checkpoint_timeout = 15min
checkpoint_completion_target = 0.9
checkpoint_warning = 600s
I got plenty of warnings that checkpoints were occurring in the range of 4-5 minutes, so I probably could have bumped up checkpoint_segments.

I would really prefer not to spend another couple of days importing, so any ideas on finishing out this import would be greatly appreciated.
Shane R.

-----Original Message-----
From: Rowley, Shane K 
Sent: Friday, December 21, 2012 11:23 AM
To: imports at openstreetmap.org
Subject: import crashed at "going over pending ways"

I ran an import of the full planet file in slim mode. It ran for 34 hours and seems to have successfully imported the data, but then crashed. Here is the end of the log of the output..

--- BEGIN LOG OUTPUT ---
Processing: Node(1686947k 46.2k/s) Way(160474k 4.76k/s) Relation(1696080 31.89/s)  parse time: 123407s

Node stats: total(1686947380), max(2050735481) in 36526s
Way stats: total(160474814), max(194569856) in 33697s
Relation stats: total(1696083), max(2619043) in 53184s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending ways...
--- END LOG OUTPUT ---

I am able to see the data in the database, so I know that part was successful.

$ psql -d planet_osm
psql (9.2.2)
Type "help" for help.

planet_osm=# select count(*) from planet_osm_point ;
  count   
----------
 40050706
(1 row)

planet_osm=# select count(*) from planet_osm_line ;
  count   
----------
 63655784
(1 row)

planet_osm=# select count(*) from planet_osm_polygon ;
  count
---------
 1197209
(1 row)

planet_osm=# select count(*) from planet_osm_roads ;
  count
---------
 6733858
(1 row)

I am wondering if it is possible to resume, or recover, or re-create the remaining steps of the import. I know the very last thing it does is to created the indexes, which I could probably do manually.

However, I don't know what it's doing when it's "going over pending ways" and "going over pending relations" (although the pending relations seemed to be an empty set the last time I got this to complete successfully).

Any suggestions? Would importing a diff clean this up? Or is that impossible until this is corrected?

Thanks for any help..
Shane R.