<div dir="ltr">Hi Kieran,<div>I added a high speed SSD and pointed the .stxxl towards that to deal with the swap slowdown. That bought me a drop in processing time from 12 hrs -> 3-4 hrs.<br></div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font face="monospace, monospace" size="1">osrm@mat4:~/osrm-backend$ cat .stxxl </font></div></div><div><div><font face="monospace, monospace" size="1"><b>disk=/mnt/tmp/stxxl,400000,syscall</b></font></div></div><div><font face="monospace, monospace" size="1"><br></font></div><div><div><font face="monospace, monospace" size="1">osrm@mat4:~/osrm-backend$ lsblk</font></div></div><div><div><font face="monospace, monospace" size="1">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT</font></div></div><div><div><font face="monospace, monospace" size="1">sda 8:0 0 223.6G 0 disk </font></div></div><div><div><font face="monospace, monospace" size="1">├─sda1 8:1 0 4G 0 part </font></div></div><div><div><font face="monospace, monospace" size="1">│ └─md0 9:0 0 4G 0 raid1 [SWAP]</font></div></div><div><div><font face="monospace, monospace" size="1">├─sda2 8:2 0 512M 0 part </font></div></div><div><div><font face="monospace, monospace" size="1">│ └─md1 9:1 0 511.7M 0 raid1 /boot</font></div></div><div><div><font face="monospace, monospace" size="1">└─sda3 8:3 0 219.1G 0 part </font></div></div><div><div><font face="monospace, monospace" size="1"> └─md2 9:2 0 219G 0 raid1 /</font></div></div><div><div><font face="monospace, monospace" size="1">sdb 8:16 0 223.6G 0 disk </font></div></div><div><div><font face="monospace, monospace" size="1">├─sdb1 8:17 0 4G 0 part </font></div></div><div><div><font face="monospace, monospace" size="1">│ └─md0 9:0 0 4G 0 raid1 [SWAP]</font></div></div><div><div><font face="monospace, monospace" size="1">├─sdb2 8:18 0 512M 0 part </font></div></div><div><div><font face="monospace, monospace" size="1">│ └─md1 9:1 0 511.7M 0 raid1 /boot</font></div></div><div><div><font face="monospace, monospace" size="1">└─sdb3 8:19 0 219.1G 0 part </font></div></div><div><div><font face="monospace, monospace" size="1"> └─md2 9:2 0 219G 0 raid1 /</font></div></div><div><div><font face="monospace, monospace" size="1"><b>sdc 8:32 0 447.1G 0 disk </b></font></div></div><div><div><font face="monospace, monospace" size="1"><b>└─sdc1 8:33 0 447.1G 0 part /mnt</b></font></div></div></blockquote><font face="monospace, monospace" size="1"><br></font><div><br></div><div><font face="monospace, monospace" size="1"><br></font></div><div><font face="monospace, monospace" size="1"><br></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 2, 2016 at 4:51 PM, Kieran Caplice <span dir="ltr"><<a href="mailto:kieran.caplice@temetra.com" target="_blank">kieran.caplice@temetra.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hello,<br>
<br>
I'm currently extracting the planet PBF (~31 GB), and it's been
running for hours. I notice in the "Running OSRM" wiki page, it says
"
On a Core i7 with 8GB RAM and (slow) 5400 RPM Samsung SATA hard
disks it took about 65 minutes to do so from a PBF formatted
planet", which is making me wonder why it's taking so long on our
server. Below are some example output messages:<br>
<br>
[info] Parsing finished after 3584.35 seconds<br>
[extractor] Erasing duplicate nodes ... ok, after 319.091s<br>
[extractor] Sorting all nodes ... ok, after 3632.87s<br>
[extractor] Building node id map ... ok, after 2025.29s<br>
[extractor] Confirming/Writing used nodes ... ok, after 1096.24s<br>
[extractor] Sorting edges by start ... ok, after 2000.08s<br>
<br>
Some stxxl errors were outputted as I set the disk size to 100GB
thinking it was enough - but I didn't think it would cause such
slowdowns as this, considering extracting the Europe PBF takes hours
also without the stxxl errors.<br>
<br>
Server specs:<br>
Ubuntu 14.04<br>
Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT)<br>
64 GB RAM @ 2133 MHz<br>
2 TB Western Digital Enterprise 7200 RPM hard drive<br>
<br>
At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).<br>
<br>
Anyone have any ideas as to what might be going on? Or is it normal
to take this long without an SSD?<br>
<br>
Thanks in advance.<br>
<div><br>
Kind regards,<br>
Kieran Caplice<br>
<br>
</div>
</div>
<br>_______________________________________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div style="font-size:small">Bjorn Madsen</div><div style="font-size:small">Director, Global Operations</div><div style="font-size:small"><br></div><div style="font-size:small"><div><span style="font-size:13px">Multi-Agent Technology Ltd.</span></div><div><div style="font-size:13px">Bessemer Drive, </div></div><div style="font-size:13px">Stevenage, </div><div style="font-size:13px">SG1 2DX</div></div><div style="font-size:small"><br></div><div style="font-size:small"><a href="mailto:bm@multiagenttechnology.com" style="color:rgb(17,85,204)" target="_blank">bm@multiagenttechnology.com</a></div><div style="font-size:small"><div>Mobile: +44 779 20 307 20</div><div>Main Office: +44 1438 310 035</div></div></div></div></div></div></div>
</div>