<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div><span style="font-size: 13.3333px;">Hi Daniel,</span></div>
<div><span style="font-size: 13.3333px;"><br>
</span></div>
Thank you for your advice.
<div><br>
</div>
<div>We are going to go the route you (and Michael, and Roland) suggested and split up the continents.</div>
<div><br>
</div>
<div><span style="font-size: 10pt;">Kind regards</span></div>
<div>Andrew</div>
<div><br>
</div>
<div><br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF327870" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b> Daniel Patterson [daniel@mapbox.com]<br>
<b>Sent:</b> 07 June 2021 04:50<br>
<b>To:</b> Mailing list to discuss Project OSRM<br>
<b>Cc:</b> Andrew King<br>
<b>Subject:</b> Re: [OSRM-talk] Computing resources<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">
<div>Andrew,</div>
<div><br>
</div>
<div> You can certainly try on a machine with virtual memory - but be warned, that can take the processing time from hours into days/weeks territory.</div>
<div><br>
</div>
<div> Unfortunately, the planet is large. If you can split it into continents and add a frontend that can route queries to an appropriate backend with continental data, you can significantly reduce the processing needs.</div>
<div><br>
</div>
<div>daniel<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Jun 6, 2021 at 2:33 PM Andrew King via OSRM-talk <<a href="mailto:osrm-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">osrm-talk@openstreetmap.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
Thank you for your help Michael.<br>
<br>
Procuring a machine with this memory size is proving quite a challenge. Is it necessary for the preprocessor to have physical memory of this size, or can virtual memory be used? I realized that the processing would be slower, but it would make it MUCH easier
to acquire a machine if it was able to make use of virtual memory to do this work.<br>
<br>
Kind regards<br>
Andrew<br>
<br>
________________________________________<br>
From: michael spreng [<a href="mailto:mailinglists@m.spreng.ch" target="_blank" rel="noopener noreferrer">mailinglists@m.spreng.ch</a>]<br>
Sent: 29 May 2021 09:20<br>
To: <a href="mailto:osrm-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">
osrm-talk@openstreetmap.org</a><br>
Subject: Re: [OSRM-talk] Computing resources<br>
<br>
Yes this is with the current version which does not have stxxl. Disk<br>
usage is about 300G per profile<br>
<br>
On 29.05.21 09:58, Andrew King wrote:<br>
> Thank you very much Michael. This is most helpful. I shall look into this when I return to work next week.<br>
><br>
> Please can you just confirm that these memory requirements that you have specified are all without STXXL (which I believe is no longer supported)?<br>
><br>
> Kind regards<br>
> Andrew<br>
><br>
><br>
> ________________________________________<br>
> From: michael spreng [<a href="mailto:mailinglists@m.spreng.ch" target="_blank" rel="noopener noreferrer">mailinglists@m.spreng.ch</a>]<br>
> Sent: 29 May 2021 08:51<br>
> To: <a href="mailto:osrm-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">
osrm-talk@openstreetmap.org</a><br>
> Subject: Re: [OSRM-talk] Computing resources<br>
><br>
> Hi<br>
><br>
> I run <a href="http://routing.openstreetmap.de" rel="noreferrer" target="_blank">
routing.openstreetmap.de</a> (which is also behind the osrm demo)<br>
><br>
> Preprocessing has a huge memory demand in the first stage, which forces<br>
> me to split the planet into three regions (Americas, Eurasia, and<br>
> Africa+Oceania) on a 256GB machine. It might be possible to process it<br>
> on 512GB, but it would be already tight, and OSM data is growing.<br>
><br>
> bike and foot need significantly more space, as there are a lot of ways<br>
> which are inaccessible to cars.<br>
><br>
> 130G of ram is needed for the car profile, and 165G for the foot or bike<br>
> profile for the server.<br>
><br>
> When splitting up the data, for each region a separate osrm-routed<br>
> instance is needed. I use a python script to dispatch the queries to<br>
> each region, see<br>
> <a href="https://github.com/fossgis-routing-server/request-by-coordinate/" rel="noreferrer" target="_blank">
https://github.com/fossgis-routing-server/request-by-coordinate/</a><br>
><br>
> Hope that helps<br>
><br>
> Michael<br>
><br>
> On 28.05.21 13:48, Andrew King via OSRM-talk wrote:<br>
>> Hi Roland,<br>
>><br>
>> Thank you so much for this useful link and advice. This is really helpful.<br>
>><br>
>> I have a couple of questions though:<br>
>><br>
>> 1. The page refers to 280 GB of STXXL disk space, however, in August<br>
>> 2020, STXXL support was removed [commit 2d0b2ca9...21f96221]. Does this<br>
>> affect those requirements at all?<br>
>><br>
>> 2. You mentioned splitting up the data. Would you then need to run two<br>
>> separate services (in your example) for each profile, or can they be<br>
>> recombined into one service after processing?<br>
>><br>
>> Many thanks<br>
>> Andrew<br>
>><br>
>><br>
>> ------------------------------------------------------------------------<br>
>> *From:* Roland Swingler via OSRM-talk [<a href="mailto:osrm-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">osrm-talk@openstreetmap.org</a>]<br>
>> *Sent:* 28 May 2021 12:14<br>
>> *To:* Mailing list to discuss Project OSRM<br>
>> *Cc:* Roland Swingler<br>
>> *Subject:* Re: [OSRM-talk] Computing resources<br>
>><br>
>> This may be useful -<br>
>> <a href="https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements" rel="noreferrer" target="_blank">
https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements</a><br>
>> <<a href="https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements" rel="noreferrer" target="_blank">https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements</a>><br>
>> although I would expect the memory requirements to be a bit higher since<br>
>> this page was written.<br>
>><br>
>> As far as I know the OSRM processes use multiple cores.<br>
>><br>
>> You may want to split up the planet file before processing if you think<br>
>> the memory requirements are too high. i.e. you're probably not going to<br>
>> have a route from Australia to Europe for example, so they could be run<br>
>> in separate instances.<br>
>><br>
>> Thanks,<br>
>> Roland<br>
>><br>
>> On Fri, 28 May 2021 at 12:08, Andrew King <<a href="mailto:andrew.king@satmap.com" target="_blank" rel="noopener noreferrer">andrew.king@satmap.com</a><br>
>> <mailto:<a href="mailto:andrew.king@satmap.com" target="_blank" rel="noopener noreferrer">andrew.king@satmap.com</a>>> wrote:<br>
>><br>
>> Hello,<br>
>><br>
>> We have recently taken on responsibility to create osrm instances<br>
>> for foot + bicycle + car routing with planet data, and we are trying<br>
>> to determine what compute specs are recommended in order to be able<br>
>> to process these.<br>
>><br>
>> Do you have this information available, or can you recommend a link<br>
>> where I can find out this information?<br>
>><br>
>> - How much memory is required/recommended to process planet data?<br>
>> - Can the osrm processes (ie osrm-extract, osrm-partition, etc) make<br>
>> use of multiple cores?<br>
>> - How long do each of the processes typically take to run on planet<br>
>> data?<br>
>><br>
>> Kind regards<br>
>> Andrew<br>
>><br>
>> _______________________________________________<br>
>> OSRM-talk mailing list<br>
>> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">OSRM-talk@openstreetmap.org</a> <mailto:<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">OSRM-talk@openstreetmap.org</a>><br>
>> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
>> <<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OSRM-talk mailing list<br>
>> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">
OSRM-talk@openstreetmap.org</a><br>
>> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">
https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
>><br>
><br>
> _______________________________________________<br>
> OSRM-talk mailing list<br>
> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">
OSRM-talk@openstreetmap.org</a><br>
> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">
https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
><br>
> _______________________________________________<br>
> OSRM-talk mailing list<br>
> <a href="mailto:OSRM-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">
OSRM-talk@openstreetmap.org</a><br>
> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">
https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
><br>
<br>
_______________________________________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
<br>
_______________________________________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank" rel="noopener noreferrer">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>