[OSRM-talk] osrm-extract taking hours to complete

[Patrick Niklaus]

Hey Kieran,


there have been a lot of structural changes (e.g. moving code from
osrm-prepare into osrm-extract) that probably invalidate that numbers.
Also we support 64bit OSM ids now, which sadly uses a lot more disk
space. I think stxxl need like 200GB. I think on our setup we have a
turn-around of 6 hours for the planet dataset on an SSD setup (car
profile, any other profile needs significantly longer). You should
probably think about updating your hard drives as this is IO bound. At
your current read/write speed it will already take more than an hour
to just write 200GB of data once. We scan it at least twice just for
pre-processing.

Cheers,
Patrick


On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice
<kieran.caplice at temetra.com> wrote:
> Hello,
>
> I'm currently extracting the planet PBF (~31 GB), and it's been running for
> hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with
> 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes
> to do so from a PBF formatted planet", which is making me wonder why it's
> taking so long on our server. Below are some example output messages:
>
> [info] Parsing finished after 3584.35 seconds
> [extractor] Erasing duplicate nodes   ... ok, after 319.091s
> [extractor] Sorting all nodes   ... ok, after 3632.87s
> [extractor] Building node id map      ... ok, after 2025.29s
> [extractor] Confirming/Writing used nodes     ... ok, after 1096.24s
> [extractor] Sorting edges by start    ... ok, after 2000.08s
>
> Some stxxl errors were outputted as I set the disk size to 100GB thinking it
> was enough - but I didn't think it would cause such slowdowns as this,
> considering extracting the Europe PBF takes hours also without the stxxl
> errors.
>
> Server specs:
> Ubuntu 14.04
> Intel Xeon CPU E5-1650 v3 @ 3.50GHz  (hex-core with HT)
> 64 GB RAM @ 2133 MHz
> 2 TB Western Digital Enterprise 7200 RPM hard drive
>
> At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).
>
> Anyone have any ideas as to what might be going on? Or is it normal to take
> this long without an SSD?
>
> Thanks in advance.
>
> Kind regards,
> Kieran Caplice
>
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>