[OSRM-talk] osrm-extract taking hours to complete

[Bjorn Madsen]

Hi Kieran,
I added a high speed SSD and pointed the .stxxl towards that to deal with
the swap slowdown. That bought me a drop in processing time from 12 hrs ->
3-4 hrs.

osrm at mat4:~/osrm-backend$ cat .stxxl
*disk=/mnt/tmp/stxxl,400000,syscall*

osrm at mat4:~/osrm-backend$ lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda       8:0    0 223.6G  0 disk
├─sda1    8:1    0     4G  0 part
│ └─md0   9:0    0     4G  0 raid1 [SWAP]
├─sda2    8:2    0   512M  0 part
│ └─md1   9:1    0 511.7M  0 raid1 /boot
└─sda3    8:3    0 219.1G  0 part
  └─md2   9:2    0   219G  0 raid1 /
sdb       8:16   0 223.6G  0 disk
├─sdb1    8:17   0     4G  0 part
│ └─md0   9:0    0     4G  0 raid1 [SWAP]
├─sdb2    8:18   0   512M  0 part
│ └─md1   9:1    0 511.7M  0 raid1 /boot
└─sdb3    8:19   0 219.1G  0 part
  └─md2   9:2    0   219G  0 raid1 /
*sdc       8:32   0 447.1G  0 disk  *
*└─sdc1    8:33   0 447.1G  0 part  /mnt*






On Wed, Mar 2, 2016 at 4:51 PM, Kieran Caplice <kieran.caplice at temetra.com>
wrote:

> Hello,
>
> I'm currently extracting the planet PBF (~31 GB), and it's been running
> for hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7
> with 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65
> minutes to do so from a PBF formatted planet", which is making me wonder
> why it's taking so long on our server. Below are some example output
> messages:
>
> [info] Parsing finished after 3584.35 seconds
> [extractor] Erasing duplicate nodes   ... ok, after 319.091s
> [extractor] Sorting all nodes   ... ok, after 3632.87s
> [extractor] Building node id map      ... ok, after 2025.29s
> [extractor] Confirming/Writing used nodes     ... ok, after 1096.24s
> [extractor] Sorting edges by start    ... ok, after 2000.08s
>
> Some stxxl errors were outputted as I set the disk size to 100GB thinking
> it was enough - but I didn't think it would cause such slowdowns as this,
> considering extracting the Europe PBF takes hours also without the stxxl
> errors.
>
> Server specs:
> Ubuntu 14.04
> Intel Xeon CPU E5-1650 v3 @ 3.50GHz  (hex-core with HT)
> 64 GB RAM @ 2133 MHz
> 2 TB Western Digital Enterprise 7200 RPM hard drive
>
> At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).
>
> Anyone have any ideas as to what might be going on? Or is it normal to
> take this long without an SSD?
>
> Thanks in advance.
>
> Kind regards,
> Kieran Caplice
>
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>


-- 
Bjorn Madsen
Director, Global Operations

Multi-Agent Technology Ltd.
Bessemer Drive,
Stevenage,
SG1 2DX

bm at multiagenttechnology.com
Mobile: +44 779 20 307 20
Main Office: +44 1438 310 035
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osrm-talk/attachments/20160302/cfec5115/attachment-0001.html>