[OSRM-talk] Computing resources

Mon Jun 7 03:50:49 UTC 2021

Andrew,

  You can certainly try on a machine with virtual memory - but be warned,
that can take the processing time from hours into days/weeks territory.

  Unfortunately, the planet is large.  If you can split it into continents
and add a frontend that can route queries to an appropriate backend with
continental data, you can significantly reduce the processing needs.

daniel

On Sun, Jun 6, 2021 at 2:33 PM Andrew King via OSRM-talk <
osrm-talk at openstreetmap.org> wrote:

> Thank you for your help Michael.
>
> Procuring a machine with this memory size is proving quite a challenge.
> Is it necessary for the preprocessor to have physical memory of this size,
> or can virtual memory be used?  I realized that the processing would be
> slower, but it would make it MUCH easier to acquire a machine if it was
> able to make use of virtual memory to do this work.
>
> Kind regards
> Andrew
>
> ________________________________________
> From: michael spreng [mailinglists at m.spreng.ch]
> Sent: 29 May 2021 09:20
> To: osrm-talk at openstreetmap.org
> Subject: Re: [OSRM-talk] Computing resources
>
> Yes this is with the current version which does not have stxxl.  Disk
> usage is about 300G per profile
>
> On 29.05.21 09:58, Andrew King wrote:
> > Thank you very much Michael. This is most helpful. I shall look into
> this when I return to work next week.
> >
> > Please can you just confirm that these memory requirements that you have
> specified are all without STXXL (which I believe is no longer supported)?
> >
> > Kind regards
> > Andrew
> >
> >
> > ________________________________________
> > From: michael spreng [mailinglists at m.spreng.ch]
> > Sent: 29 May 2021 08:51
> > To: osrm-talk at openstreetmap.org
> > Subject: Re: [OSRM-talk] Computing resources
> >
> > Hi
> >
> > I run routing.openstreetmap.de (which is also behind the osrm demo)
> >
> > Preprocessing has a huge memory demand in the first stage, which forces
> > me to split the planet into three regions (Americas, Eurasia, and
> > Africa+Oceania) on a 256GB machine. It might be possible to process it
> > on 512GB, but it would be already tight, and OSM data is growing.
> >
> > bike and foot need significantly more space, as there are a lot of ways
> > which are inaccessible to cars.
> >
> > 130G of ram is needed for the car profile, and 165G for the foot or bike
> > profile for the server.
> >
> > When splitting up the data, for each region a separate osrm-routed
> > instance is needed. I use a python script to dispatch the queries to
> > each region, see
> > https://github.com/fossgis-routing-server/request-by-coordinate/
> >
> > Hope that helps
> >
> > Michael
> >
> > On 28.05.21 13:48, Andrew King via OSRM-talk wrote:
> >> Hi Roland,
> >>
> >> Thank you so much for this useful link and advice. This is really
> helpful.
> >>
> >> I have a couple of questions though:
> >>
> >> 1. The page refers to 280 GB of STXXL disk space, however, in August
> >> 2020, STXXL support was removed [commit 2d0b2ca9...21f96221]. Does this
> >> affect those requirements at all?
> >>
> >> 2. You mentioned splitting up the data. Would you then need to run two
> >> separate services (in your example) for each profile, or can they be
> >> recombined into one service after processing?
> >>
> >> Many thanks
> >> Andrew
> >>
> >>
> >> ------------------------------------------------------------------------
> >> *From:* Roland Swingler via OSRM-talk [osrm-talk at openstreetmap.org]
> >> *Sent:* 28 May 2021 12:14
> >> *To:* Mailing list to discuss Project OSRM
> >> *Cc:* Roland Swingler
> >> *Subject:* Re: [OSRM-talk] Computing resources
> >>
> >> This may be useful -
> >>
> https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements
> >> <
> https://github.com/Project-OSRM/osrm-backend/wiki/Disk-and-Memory-Requirements
> >
> >> although I would expect the memory requirements to be a bit higher since
> >> this page was written.
> >>
> >> As far as I know the OSRM processes use multiple cores.
> >>
> >> You may want to split up the planet file before processing if you think
> >> the memory requirements are too high. i.e. you're probably not going to
> >> have a route from Australia to Europe for example, so they could be run
> >> in separate instances.
> >>
> >> Thanks,
> >> Roland
> >>
> >> On Fri, 28 May 2021 at 12:08, Andrew King <andrew.king at satmap.com
> >> <mailto:andrew.king at satmap.com>> wrote:
> >>
> >>     Hello,
> >>
> >>     We have recently taken on responsibility to create osrm instances
> >>     for foot + bicycle + car routing with planet data, and we are trying
> >>     to determine what compute specs are recommended in order to be able
> >>     to process these.
> >>
> >>     Do you have this information available, or can you recommend a link
> >>     where I can find out this information?
> >>
> >>     - How much memory is required/recommended to process planet data?
> >>     - Can the osrm processes (ie osrm-extract, osrm-partition, etc) make
> >>     use of multiple cores?
> >>     - How long do each of the processes typically take to run on planet
> >>     data?
> >>
> >>     Kind regards
> >>     Andrew
> >>
> >>     _______________________________________________
> >>     OSRM-talk mailing list
> >>     OSRM-talk at openstreetmap.org <mailto:OSRM-talk at openstreetmap.org>
> >>     https://lists.openstreetmap.org/listinfo/osrm-talk
> >>     <https://lists.openstreetmap.org/listinfo/osrm-talk>
> >>
> >>
> >> _______________________________________________
> >> OSRM-talk mailing list
> >> OSRM-talk at openstreetmap.org
> >> https://lists.openstreetmap.org/listinfo/osrm-talk
> >>
> >
> > _______________________________________________
> > OSRM-talk mailing list
> > OSRM-talk at openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >
> > _______________________________________________
> > OSRM-talk mailing list
> > OSRM-talk at openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/osrm-talk
> >
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osrm-talk/attachments/20210606/7314f368/attachment.htm>