[Tile-serving] [osm2pgsql] Import 'Killed' twice now while 'Going over pending ways...' (#218)

Mon Dec 22 21:13:55 UTC 2014

I had time again last week to kick off another run, this time with 18 GB of RAM used. It hasn't been killed due to low memory yet, but is not really giving any feedback and does not appear to be processing much of anything.

    top - 21:06:43 up 42 days,  6:21,  1 user,  load average: 0.00, 0.01, 0.05
    Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem:  24109800 total, 23914420 used,   195380 free,    22332 buffers
    KiB Swap:    49148 total,    49148 used,        0 free. 23072972 cached Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    14230 www-data  20   0 2108916  13592    648 S   0.3  0.1   0:29.09 apache2
    1 root      20   0   37080   5120    260 S   0.0  0.0   0:02.70 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.55 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.84 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0  10:03.38 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   2:34.15 rcuos/0
    9 root      20   0       0      0      0 S   0.0  0.0   3:33.02 rcuos/1
   10 root      20   0       0      0      0 S   0.0  0.0   1:19.30 rcuos/2
   11 root      20   0       0      0      0 S   0.0  0.0   0:47.87 rcuos/3
   12 root      20   0       0      0      0 S   0.0  0.0   0:33.98 rcuos/4
   13 root      20   0       0      0      0 S   0.0  0.0   0:33.67 rcuos/5
   14 root      20   0       0      0      0 S   0.0  0.0   0:19.81 rcuos/6
   15 root      20   0       0      0      0 S   0.0  0.0   0:21.05 rcuos/7
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0
   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2
   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3
   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/4
   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/5
   23 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/6
   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/7

Oops, I guess that was wrong. I hit enter in the tmux that it was running in and got the 'Killed' message.

    /s) Relation(2948500 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s)         Relation(2948510 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s)     Relation(2948520 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948530 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948540 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948550 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948560 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948570 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948580 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948590 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948600 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948610 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948620 16.Processing: Node(2622685k 303.7k/s
 ) Way(261934k 1.40k/s) Relation(2948630 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948640 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948650 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948660 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948670 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948680 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948690 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948700 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948710 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948720 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948730 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948740 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948750 16.Processing: Node(2622685k 303.
 7k/s) Way(261934k 1.40k/s) Relation(2948760 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948770 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948780 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948790 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948800 16.Processing: Node(2622685k 303.7k/s) Way(261934k 1.40k/s) Relation(2948810 16.21/s)  parse time: 377054s

    Node stats: total(2622685082), max(3205477953) in 8635s
    Way stats: total(261934287), max(314500881) in 186526s
    Relation stats: total(2948815), max(4226619) in 181893s
    Committing transaction for planet_osm_point
    Committing transaction for planet_osm_line
    Committing transaction for planet_osm_polygon
    Committing transaction for planet_osm_roads

    Going over pending ways...
    ubuntu at map-server:/map-data$
    [1]+  Killed                  osm2pgsql --slim -C 18000 --number-processes 7 planet-141126.osm.pbf
    ubuntu at map-server:/map-data$

I guess that I'll have to just try something like a North America import for now.

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/218#issuecomment-67890495
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20141222/320b402d/attachment-0001.html>