<div dir="ltr"><div>Andrew,</div><div><br></div><div> You can certainly try on a machine with virtual memory - but be warned, that can take the processing time from hours into days/weeks territory.</div><div><br></div><div> Unfortunately, the planet is large. If you can split it into continents and add a frontend that can route queries to an appropriate backend with continental data, you can significantly reduce the processing needs.</div><div><br></div><div>daniel<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 6, 2021 at 2:33 PM Andrew King via OSRM-talk <<a href="mailto:osrm-talk@openstreetmap.org">osrm-talk@openstreetmap.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thank you for your help Michael.<br>
<br>
Procuring a machine with this memory size is proving quite a challenge. Is it necessary for the preprocessor to have physical memory of this size, or can virtual memory be used? I realized that the processing would be slower, but it would make it MUCH easier to acquire a machine if it was able to make use of virtual memory to do this work.<br>
<br>
Kind regards<br>
Andrew<br>
<br>
________________________________________<br>
From: michael spreng [<a href="mailto:mailinglists@m.spreng.ch" target="_blank">mailinglists@m.spreng.ch</a>]<br>
Sent: 29 May 2021 09:20<br>
To: <a href="mailto:osrm-talk@openstreetmap.org" target="_blank">osrm-talk@openstreetmap.org</a><br>
Subject: Re: [OSRM-talk] Computing resources<br>
<br>
Yes this is with the current version which does not have stxxl. Disk<br>
usage is about 300G per profile<br>
<br>
On 29.05.21 09:58, Andrew King wrote:<br>
> Thank you very much Michael. This is most helpful. I shall look into this when I return to work next week.<br>
><br>
> Please can you just confirm that these memory requirements that you have specified are all without STXXL (which I believe is no longer supported)?<br>
><br>
> Kind regards<br>
> Andrew<br>
><br>
><br>
> ________________________________________<br>
> From: michael spreng [<a href="mailto:mailinglists@m.spreng.ch" target="_blank">mailinglists@m.spreng.ch</a>]<br>
> Sent: 29 May 2021 08:51<br>
> To: <a href="mailto:osrm-talk@openstreetmap.org" target="_blank">osrm-talk@openstreetmap.org</a><br>
> Subject: Re: [OSRM-talk] Computing resources<br>
><br>
> Hi<br>
><br>
> I run <a href="http://routing.openstreetmap.de" rel="noreferrer" target="_blank">routing.openstreetmap.de</a> (which is also behind the osrm demo)<br>
><br>
> Preprocessing has a huge memory demand in the first stage, which forces<br>
> me to split the planet into three regions (Americas, Eurasia, and<br>
> Africa+Oceania) on a 256GB machine. It might be possible to process it<br>
> on 512GB, but it would be already tight, and OSM data is growing.<br>
><br>
> bike and foot need significantly more space, as there are a lot of ways<br>
> which are inaccessible to cars.<br>
><br>
> 130G of ram is needed for the car profile, and 165G for the foot or bike<br>
> profile for the server.<br>
><br>
> When splitting up the data, for each region a separate osrm-routed<br>
> instance is needed. I use a python script to dispatch the queries to<br>
> each region, see<br>
> <a href="https://github.com/fossgis-routing-server/request-by-coordinate/" rel="noreferrer" target="_blank">https://github.com/fossgis-routing-server/request-by-coordinate/</a><br>
><br>
> Hope that helps<br>
><br>
> Michael<br>
><br>
> On 28.05.21 13:48, Andrew King via OSRM-talk wrote:<br>
>> Hi Roland,<br>
>><br>
>> Thank you so much for this useful link and advice. This is really helpful.<br>
>><br>
>> I have a couple of questions though:<br>
>><br>
>> 1. The page refers to 280 GB of STXXL disk space, however, in August<br>
>> 2020, STXXL support was removed [commit 2d0b2ca9...21f96221]. Does this<br>
>> affect those requirements at all?<br>
>><br>
>> 2. You mentioned splitting up the data. Would you then need to run two<br>
>> separate services (in your example) for each profile, or can they be<br>
>> recombined into one service after processing?<br>
>><br>
>> Many thanks<br>
>> Andrew<br>
>><br>
>><br>
>> ------------------------------------------------------------------------<br>
>> *From:* Roland Swingler via OSRM-talk [<a href="mailto:osrm-talk@openstreetmap.org" target="_blank">osrm-talk@openstreetmap.org</a>]<br>
>> *Sent:* 28 May 2021 12:14<br>
>> *To:* Mailing list to discuss Project OSRM<br>
>> *Cc:* Roland Swingler<br>
>> *Subject:* Re: [OSRM-talk] Computing resources<br>
>><br>
>> This may be useful -<br>
>> <a href="https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements" rel="noreferrer" target="_blank">https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements</a><br>
>> <<a href="https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements" rel="noreferrer" target="_blank">https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements</a>><br>
>> although I would expect the memory requirements to be a bit higher since<br>
>> this page was written.<br>
>><br>
>> As far as I know the OSRM processes use multiple cores.<br>
>><br>
>> You may want to split up the planet file before processing if you think<br>
>> the memory requirements are too high. i.e. you're probably not going to<br>
>> have a route from Australia to Europe for example, so they could be run<br>
>> in separate instances.<br>
>><br>
>> Thanks,<br>
>> Roland<br>
>><br>
>> On Fri, 28 May 2021 at 12:08, Andrew King <<a href="mailto:andrew.king@satmap.com" target="_blank">andrew.king@satmap.com</a><br>
>> <mailto:<a href="mailto:andrew.king@satmap.com" target="_blank">andrew.king@satmap.com</a>>> wrote:<br>
>><br>
>> Hello,<br>
>><br>
>> We have recently taken on responsibility to create osrm instances<br>
>> for foot + bicycle + car routing with planet data, and we are trying<br>
>> to determine what compute specs are recommended in order to be able<br>
>> to process these.<br>
>><br>
>> Do you have this information available, or can you recommend a link<br>
>> where I can find out this information?<br>
>><br>
>> - How much memory is required/recommended to process planet data?<br>
>> - Can the osrm processes (ie osrm-extract, osrm-partition, etc) make<br>
>> use of multiple cores?<br>
>> - How long do each of the processes typically take to run on planet<br>
>> data?<br>
>><br>
>> Kind regards<br>
>> Andrew<br>
>><br>
>> _______________________________________________<br>
>> OSRM-talk mailing list<br>
>> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a> <mailto:<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a>><br>
>> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
>> <<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OSRM-talk mailing list<br>
>> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
>> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
>><br>
><br>
> _______________________________________________<br>
> OSRM-talk mailing list<br>
> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
><br>
> _______________________________________________<br>
> OSRM-talk mailing list<br>
> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
><br>
<br>
_______________________________________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
<br>
_______________________________________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
</blockquote></div>