[OSRM-talk] Computing resources

Tue Jun 8 12:19:37 UTC 2021

Hi Daniel,

Thank you for your advice.

We are going to go the route you (and Michael, and Roland) suggested and split up the continents.

Kind regards
Andrew

________________________________
From: Daniel Patterson [daniel at mapbox.com]
Sent: 07 June 2021 04:50
To: Mailing list to discuss Project OSRM
Cc: Andrew King
Subject: Re: [OSRM-talk] Computing resources

Andrew,

  You can certainly try on a machine with virtual memory - but be warned, that can take the processing time from hours into days/weeks territory.

  Unfortunately, the planet is large.  If you can split it into continents and add a frontend that can route queries to an appropriate backend with continental data, you can significantly reduce the processing needs.

daniel

On Sun, Jun 6, 2021 at 2:33 PM Andrew King via OSRM-talk <osrm-talk at openstreetmap.org<mailto:osrm-talk at openstreetmap.org>> wrote:
Thank you for your help Michael.

Procuring a machine with this memory size is proving quite a challenge.  Is it necessary for the preprocessor to have physical memory of this size, or can virtual memory be used?  I realized that the processing would be slower, but it would make it MUCH easier to acquire a machine if it was able to make use of virtual memory to do this work.

Kind regards
Andrew

________________________________________
From: michael spreng [mailinglists at m.spreng.ch<mailto:mailinglists at m.spreng.ch>]
Sent: 29 May 2021 09:20
To: osrm-talk at openstreetmap.org<mailto:osrm-talk at openstreetmap.org>
Subject: Re: [OSRM-talk] Computing resources

Yes this is with the current version which does not have stxxl.  Disk
usage is about 300G per profile

On 29.05.21 09:58, Andrew King wrote:
> Thank you very much Michael. This is most helpful. I shall look into this when I return to work next week.
>
> Please can you just confirm that these memory requirements that you have specified are all without STXXL (which I believe is no longer supported)?
>
> Kind regards
> Andrew
>
>
> ________________________________________
> From: michael spreng [mailinglists at m.spreng.ch<mailto:mailinglists at m.spreng.ch>]
> Sent: 29 May 2021 08:51
> To: osrm-talk at openstreetmap.org<mailto:osrm-talk at openstreetmap.org>
> Subject: Re: [OSRM-talk] Computing resources
>
> Hi
>
> I run routing.openstreetmap.de<http://routing.openstreetmap.de> (which is also behind the osrm demo)
>
> Preprocessing has a huge memory demand in the first stage, which forces
> me to split the planet into three regions (Americas, Eurasia, and
> Africa+Oceania) on a 256GB machine. It might be possible to process it
> on 512GB, but it would be already tight, and OSM data is growing.
>
> bike and foot need significantly more space, as there are a lot of ways
> which are inaccessible to cars.
>
> 130G of ram is needed for the car profile, and 165G for the foot or bike
> profile for the server.
>
> When splitting up the data, for each region a separate osrm-routed
> instance is needed. I use a python script to dispatch the queries to
> each region, see
> https://github.com/fossgis-routing-server/request-by-coordinate/
>
> Hope that helps
>
> Michael
>
> On 28.05.21 13:48, Andrew King via OSRM-talk wrote:
>> Hi Roland,
>>
>> Thank you so much for this useful link and advice. This is really helpful.
>>
>> I have a couple of questions though:
>>
>> 1. The page refers to 280 GB of STXXL disk space, however, in August
>> 2020, STXXL support was removed [commit 2d0b2ca9...21f96221]. Does this
>> affect those requirements at all?
>>
>> 2. You mentioned splitting up the data. Would you then need to run two
>> separate services (in your example) for each profile, or can they be
>> recombined into one service after processing?
>>
>> Many thanks
>> Andrew
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Roland Swingler via OSRM-talk [osrm-talk at openstreetmap.org<mailto:osrm-talk at openstreetmap.org>]
>> *Sent:* 28 May 2021 12:14
>> *To:* Mailing list to discuss Project OSRM
>> *Cc:* Roland Swingler
>> *Subject:* Re: [OSRM-talk] Computing resources
>>
>> This may be useful -
>> https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements
>> <https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements>
>> although I would expect the memory requirements to be a bit higher since
>> this page was written.
>>
>> As far as I know the OSRM processes use multiple cores.
>>
>> You may want to split up the planet file before processing if you think
>> the memory requirements are too high. i.e. you're probably not going to
>> have a route from Australia to Europe for example, so they could be run
>> in separate instances.
>>
>> Thanks,
>> Roland
>>
>> On Fri, 28 May 2021 at 12:08, Andrew King <andrew.king at satmap.com<mailto:andrew.king at satmap.com>
>> <mailto:andrew.king at satmap.com<mailto:andrew.king at satmap.com>>> wrote:
>>
>>     Hello,
>>
>>     We have recently taken on responsibility to create osrm instances
>>     for foot + bicycle + car routing with planet data, and we are trying
>>     to determine what compute specs are recommended in order to be able
>>     to process these.
>>
>>     Do you have this information available, or can you recommend a link
>>     where I can find out this information?
>>
>>     - How much memory is required/recommended to process planet data?
>>     - Can the osrm processes (ie osrm-extract, osrm-partition, etc) make
>>     use of multiple cores?
>>     - How long do each of the processes typically take to run on planet
>>     data?
>>
>>     Kind regards
>>     Andrew
>>
>>     _______________________________________________
>>     OSRM-talk mailing list
>>     OSRM-talk at openstreetmap.org<mailto:OSRM-talk at openstreetmap.org> <mailto:OSRM-talk at openstreetmap.org<mailto:OSRM-talk at openstreetmap.org>>
>>     https://lists.openstreetmap.org/listinfo/osrm-talk
>>     <https://lists.openstreetmap.org/listinfo/osrm-talk>
>>
>>
>> _______________________________________________
>> OSRM-talk mailing list
>> OSRM-talk at openstreetmap.org<mailto:OSRM-talk at openstreetmap.org>
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org<mailto:OSRM-talk at openstreetmap.org>
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org<mailto:OSRM-talk at openstreetmap.org>
> https://lists.openstreetmap.org/listinfo/osrm-talk
>

_______________________________________________
OSRM-talk mailing list
OSRM-talk at openstreetmap.org<mailto:OSRM-talk at openstreetmap.org>
https://lists.openstreetmap.org/listinfo/osrm-talk

_______________________________________________
OSRM-talk mailing list
OSRM-talk at openstreetmap.org<mailto:OSRM-talk at openstreetmap.org>
https://lists.openstreetmap.org/listinfo/osrm-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osrm-talk/attachments/20210608/df02c872/attachment.htm>