[OSRM-talk] osrm-extract taking hours to complete

[Kieran Caplice]

Hi Patrick,

That makes sense then. It's obvious the process is just going to take 
upwards of 8-10 hours for us in that case.

Thanks for the help.

Kind regards,
Kieran Caplice

On 02/03/16 17:01, Patrick Niklaus wrote:
> Hey Kieran,
>
>
> there have been a lot of structural changes (e.g. moving code from
> osrm-prepare into osrm-extract) that probably invalidate that numbers.
> Also we support 64bit OSM ids now, which sadly uses a lot more disk
> space. I think stxxl need like 200GB. I think on our setup we have a
> turn-around of 6 hours for the planet dataset on an SSD setup (car
> profile, any other profile needs significantly longer). You should
> probably think about updating your hard drives as this is IO bound. At
> your current read/write speed it will already take more than an hour
> to just write 200GB of data once. We scan it at least twice just for
> pre-processing.
>
> Cheers,
> Patrick
>
>
> On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice
> <kieran.caplice at temetra.com> wrote:
>> Hello,
>>
>> I'm currently extracting the planet PBF (~31 GB), and it's been running for
>> hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with
>> 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes
>> to do so from a PBF formatted planet", which is making me wonder why it's
>> taking so long on our server. Below are some example output messages:
>>
>> [info] Parsing finished after 3584.35 seconds
>> [extractor] Erasing duplicate nodes   ... ok, after 319.091s
>> [extractor] Sorting all nodes   ... ok, after 3632.87s
>> [extractor] Building node id map      ... ok, after 2025.29s
>> [extractor] Confirming/Writing used nodes     ... ok, after 1096.24s
>> [extractor] Sorting edges by start    ... ok, after 2000.08s
>>
>> Some stxxl errors were outputted as I set the disk size to 100GB thinking it
>> was enough - but I didn't think it would cause such slowdowns as this,
>> considering extracting the Europe PBF takes hours also without the stxxl
>> errors.
>>
>> Server specs:
>> Ubuntu 14.04
>> Intel Xeon CPU E5-1650 v3 @ 3.50GHz  (hex-core with HT)
>> 64 GB RAM @ 2133 MHz
>> 2 TB Western Digital Enterprise 7200 RPM hard drive
>>
>> At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).
>>
>> Anyone have any ideas as to what might be going on? Or is it normal to take
>> this long without an SSD?
>>
>> Thanks in advance.
>>
>> Kind regards,
>> Kieran Caplice
>>
>>
>> _______________________________________________
>> OSRM-talk mailing list
>> OSRM-talk at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk