[OSRM-talk] osrm-extract taking hours to complete
kieran.caplice at temetra.com
Wed Mar 2 17:23:19 UTC 2016
That makes sense then. It's obvious the process is just going to take
upwards of 8-10 hours for us in that case.
Thanks for the help.
On 02/03/16 17:01, Patrick Niklaus wrote:
> Hey Kieran,
> there have been a lot of structural changes (e.g. moving code from
> osrm-prepare into osrm-extract) that probably invalidate that numbers.
> Also we support 64bit OSM ids now, which sadly uses a lot more disk
> space. I think stxxl need like 200GB. I think on our setup we have a
> turn-around of 6 hours for the planet dataset on an SSD setup (car
> profile, any other profile needs significantly longer). You should
> probably think about updating your hard drives as this is IO bound. At
> your current read/write speed it will already take more than an hour
> to just write 200GB of data once. We scan it at least twice just for
> On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice
> <kieran.caplice at temetra.com> wrote:
>> I'm currently extracting the planet PBF (~31 GB), and it's been running for
>> hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with
>> 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes
>> to do so from a PBF formatted planet", which is making me wonder why it's
>> taking so long on our server. Below are some example output messages:
>> [info] Parsing finished after 3584.35 seconds
>> [extractor] Erasing duplicate nodes ... ok, after 319.091s
>> [extractor] Sorting all nodes ... ok, after 3632.87s
>> [extractor] Building node id map ... ok, after 2025.29s
>> [extractor] Confirming/Writing used nodes ... ok, after 1096.24s
>> [extractor] Sorting edges by start ... ok, after 2000.08s
>> Some stxxl errors were outputted as I set the disk size to 100GB thinking it
>> was enough - but I didn't think it would cause such slowdowns as this,
>> considering extracting the Europe PBF takes hours also without the stxxl
>> Server specs:
>> Ubuntu 14.04
>> Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT)
>> 64 GB RAM @ 2133 MHz
>> 2 TB Western Digital Enterprise 7200 RPM hard drive
>> At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).
>> Anyone have any ideas as to what might be going on? Or is it normal to take
>> this long without an SSD?
>> Thanks in advance.
>> Kind regards,
>> Kieran Caplice
>> OSRM-talk mailing list
>> OSRM-talk at openstreetmap.org
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
More information about the OSRM-talk