[OSRM-talk] osrm-extract taking hours to complete

Kieran Caplice kieran.caplice at temetra.com
Wed Mar 2 17:25:44 UTC 2016

Hi Patrick,

Just wanted to ping you off-list. I just had a question about your 
update process....how often if at all do you update your PBF data, and 
how do you manage the process? We're thinking of just setting a cron job 
to download and extract the new PBF every weekend to a temporary folder, 
stop the server, move the new data to the correct location and start the 
server again.

Kind regards,
Kieran Caplice

On 02/03/16 17:01, Patrick Niklaus wrote:
> Hey Kieran,
> there have been a lot of structural changes (e.g. moving code from
> osrm-prepare into osrm-extract) that probably invalidate that numbers.
> Also we support 64bit OSM ids now, which sadly uses a lot more disk
> space. I think stxxl need like 200GB. I think on our setup we have a
> turn-around of 6 hours for the planet dataset on an SSD setup (car
> profile, any other profile needs significantly longer). You should
> probably think about updating your hard drives as this is IO bound. At
> your current read/write speed it will already take more than an hour
> to just write 200GB of data once. We scan it at least twice just for
> pre-processing.
> Cheers,
> Patrick
> On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice
> <kieran.caplice at temetra.com> wrote:
>> Hello,
>> I'm currently extracting the planet PBF (~31 GB), and it's been running for
>> hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with
>> 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes
>> to do so from a PBF formatted planet", which is making me wonder why it's
>> taking so long on our server. Below are some example output messages:
>> [info] Parsing finished after 3584.35 seconds
>> [extractor] Erasing duplicate nodes   ... ok, after 319.091s
>> [extractor] Sorting all nodes   ... ok, after 3632.87s
>> [extractor] Building node id map      ... ok, after 2025.29s
>> [extractor] Confirming/Writing used nodes     ... ok, after 1096.24s
>> [extractor] Sorting edges by start    ... ok, after 2000.08s
>> Some stxxl errors were outputted as I set the disk size to 100GB thinking it
>> was enough - but I didn't think it would cause such slowdowns as this,
>> considering extracting the Europe PBF takes hours also without the stxxl
>> errors.
>> Server specs:
>> Ubuntu 14.04
>> Intel Xeon CPU E5-1650 v3 @ 3.50GHz  (hex-core with HT)
>> 64 GB RAM @ 2133 MHz
>> 2 TB Western Digital Enterprise 7200 RPM hard drive
>> At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).
>> Anyone have any ideas as to what might be going on? Or is it normal to take
>> this long without an SSD?
>> Thanks in advance.
>> Kind regards,
>> Kieran Caplice
>> _______________________________________________
>> OSRM-talk mailing list
>> OSRM-talk at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk

More information about the OSRM-talk mailing list