[OSM-dev] osm2pgsql, planet-110126.osm, doubts

Juan Lucas Domínguez Rubio juan_lucas_dr at yahoo.com
Wed Feb 9 11:13:51 GMT 2011


Dear list,

I'm in a rather weird situation (mostly my fault). These are the facts:

- I have launched the import of the OSM planet like this:

osm2pgsql -d osmdb -l --slim -U postgres -W -H 127.0.0.1 -v /mnt/data/planet-110126.osm &

- The size of planet-110126.osm is exactly 216,176,647,801 bytes

- I didn't redirect the output and I closed the session, so I cannot see the counters (I mean the ones that say "ways 200K", etc). The process is running since last Friday on a Ubuntu 10.10 machine. I suspect RAM size is the bottleneck. 'free -m says':

             total       used       free     shared    buffers     cached
Mem:          5983       5929         54          0         21       4529
-/+ buffers/cache:       1378       4605
Swap:         5883          0       5883


'top' says:

=========================================
top - 11:59:59 up 5 days, 19:32,  2 users,  load average: 1.01, 1.04, 1.06
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.7%us,  0.9%sy,  0.0%ni, 77.9%id, 17.0%wa,  0.0%hi,  0.5%si,  0.0%st
Mem:   6127440k total,  6083552k used,    43888k free,    21088k buffers
Swap:  6025212k total,        0k used,  6025212k free,  4648332k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1844 postgres  20   0  208m 134m 131m S   15  2.2 911:41.53 postgres
 1838 desa      20   0 1055m 910m 1292 S    6 15.2 641:52.10 osm2pgsql
 1845 postgres  20   0  218m 142m 132m D    1  2.4  41:05.37 postgres
   47 root      20   0     0    0    0 S    0  0.0   5:06.97 kswapd0
 1841 postgres  20   0  253m 161m 132m S    0  2.7  36:31.77 postgres
 1842 postgres  20   0  287m 188m 132m S    0  3.2   9:52.45 postgres
 4669 postgres  20   0 19276 1324  968 R    0  0.0   0:00.02 top
    1 root      20   0 23740  808  212 S    0  0.0   0:01.56 init
...
=========================================

I'm a bit worried about the %MEM = 15.2 (?). 

Is there a way to see again the output? I think 'fg' won't work as 'job' does not list it because I didn't use CTRL+Z and 'bg'.

If I do cat /proc/1828/10 (1828 being the process ID), I get:

rchar: 216047549356
wchar: 42604
syscr: 52746017
syscw: 175423
read_bytes: 216274485248
write_bytes: 0
cancelled_write_bytes: 0

As you can see the number of bytes read is already above the size of the .osm file (maybe osm2pgsql has created and read some temporary files?) Anyway I think the app has already read more than 95% of the .osm file. What do you think?

What worries me is that if I access the DB from pgAdmin, it says it has no tables apart from the default tables of every postgis DB.

What do you think?
Is it normal that osm2pgsql has read more than 200 GB and no table appears in the DB? Are there (invisible?) temporary tables somewhere?

Which % of the job you think has been done so far?
Maybe I should kill the process and restart the import again?


Thanks,
Juan Lucas


 
____________________________________________________________________________________
Now that's room service!  Choose from over 150,000 hotels
in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097



More information about the dev mailing list