[OSRM-talk] Pre-procesing planet.pbf with 128Gb RAM

Daniel Patterson daniel at mapbox.com
Tue Mar 6 04:28:31 UTC 2018


Hi Jose,

  That's correct, the published node binaries aren't built with STXXL
support.

  You can always use swap, but due to the memory access patterns that OSRM
uses, it tends to be *incredibly* slow (what takes hours in RAM can take
days or weeks with swap, depending on the amount).  STXXL is greatly
preferred when you're constrained for RAM.

  The more RAM you can supply, the faster things will go.

  These numbers are a bit out of date, but will give you a sense of what
you'll need:
https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements

  I haven't run the whole planet for a while (here at Mapbox, we divide it
up into the large disconnected geographic regions), so I don't have any
more up-to-date numbers than that wiki page.

danie

On Mon, Mar 5, 2018 at 7:28 PM, Jose Florido <joseflorido at gmail.com> wrote:

> Thanks!  So that means we cannot use the provided node binaries, right?
>
> Do you think that 192 gb of RAM + swap would be enough to be able to do
> preparation without STXXL enabled?
>
> Thanks!
> Jose
>
>
> On Mon, Mar 5, 2018, 19:09 Daniel Patterson <daniel at mapbox.com> wrote:
>
>> Hi Jose,
>>
>>   128GB is not enough to do *everything* in RAM - over 300GB of data is
>> generated during some phases.
>>
>>   You'll probably want to do a custom build of OSRM and enable STXXL with
>> `cmake -DENABLE_STXXL=ON ..`.  This will build OSRM with the ability to do
>> optimized data-to-disk swapping using the "libstxxl" library, which is
>> faster than just using regular swap space.
>>
>> daniel
>>
>> On Mon, Mar 5, 2018 at 11:44 AM, Jose Florido <joseflorido at gmail.com>
>> wrote:
>>
>>> Hi!
>>>
>>> Is it possible to pre-process the planet file with OSRM 5.16.2 and
>>> only 128Gb ram? We do have a large SSD disk and pre processing doesn’t
>>> need to be super fast.
>>>
>>> Thanks!
>>> Jose
>>>
>>> _______________________________________________
>>> OSRM-talk mailing list
>>> OSRM-talk at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>>
>>
>> _______________________________________________
>> OSRM-talk mailing list
>> OSRM-talk at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osrm-talk/attachments/20180305/5736b2e9/attachment.html>


More information about the OSRM-talk mailing list