[OSRM-talk] Road speeds and profile restrictions

Thu Oct 15 15:38:16 UTC 2015

Richard,

  For processing larger data sets quickly, consider using pay-by-the-hour computing resources, like Amazon EC2.
  Using a high-memory EC2 machine should be able to get EU processed in just a couple of hours.

  As long as the architecture is the same as the machine you will run `osrm-routed` on, you should be able to
  copy the processed data files onto your target machine.  Actual routing requires a lot less memory than processing.

daniel

> On Oct 15, 2015, at 6:43 AM, Richard Marsden <winwaed at gmail.com> wrote:
> 
> Thanks - I'll try to work through the profiles - hopefully they'll
> make some sense as I work through them.
> 
> osrm-prepare also took two weeks to crash.Perhaps it did have a sudden
> spike in memory usage before crashing - it was overnight, so I don't
> know for sure. Anyway I'm not in a rush to try another two week run.
> 
> Cheers,
> Richard
> 
> On Wed, Oct 14, 2015 at 4:20 AM, Daniel Hofmann <hofmann at mapbox.com> wrote:
>> If osrm-prepare crashed, just add more swap. Also make sure to read this
>> ticket:
>>> https://github.com/Project-OSRM/osrm-backend/issues/1698
>> 
>> I don't think we have documentation for the profiles. I know they are
>> somewhat ugly and huge. I would recommend taking a look at the profile's
>> general structure first, so that you roughly understand e.g. that there are
>> callbacks for OSM nodes and ways, and what they do. After that I would
>> simply trace the globals through the script. That is e.g. search for all
>> places where a certain whitelist is used and build a mental model based on
>> that. Pen and paper may help :)
>> 
>>> https://github.com/Project-OSRM/osrm-backend/blob/develop/profiles/car.lua
>> 
>> You can find the C++ side of things in the extractor directory:
>>> https://github.com/Project-OSRM/osrm-backend/tree/develop/extractor
>> 
>> For example, the scripting environment uses luabind to make a few functions
>> visible, e.g. the get_value_by_key function, that then can be used in the
>> profiles. The other way, that is using lua functions in C++ is mostly done
>> in the extractor implementation: I would recommend just searching for the
>> symbols you want to know more about:
>> 
>>> 
>>> https://github.com/Project-OSRM/osrm-backend/search?utf8=%E2%9C%93&q=node_function&type=Code
>> 
>> Cheers,
>> Daniel
>> 
>> On Mon, Oct 12, 2015 at 4:00 AM, Richard Marsden <winwaed at gmail.com> wrote:
>>> 
>>> It is possible I've just managed to demonstrate this to myself.
>>> osrm-prepare crashed out on the Europe-wide dataset  at the 90+%
>>> complete level with 30GB swap (16GB real). For much of the run it
>>> didn't go above about 15GB swap. Very slow of course - but that was to
>>> be expected.
>>> 
>>> This continues to be a side / evaluation project, but I'm continuing
>>> to investigate. Looks like the PC in question should be able to handle
>>> most countries, but will balk at most continents, and the very largest
>>> countries (e.g. the US). High-RAM hardware is relatively pricey, but
>>> it is a possibility in the new year.
>>> 
>>> Looking at the individual pieces, next I'm going to look at the
>>> profile.lua file. I've never used LUA before, but that is easy enough
>>> to remedy with online tutorials and the various books that are
>>> available. But what about the script that is used by OSRM? Is there
>>> any documentation as to the variables and/or functions that are
>>> expected?  And the tags used in OSM that the script is expected to
>>> map?
>>> 
>>> 
>>> Cheers,
>>> 
>>> Richard Marsden
>>> 
>>> On Thu, Sep 17, 2015 at 4:37 PM, Patrick Niklaus
>>> <patrick.niklaus at student.kit.edu> wrote:
>>>> W.r.t. the pre-preprocessing you are correct.
>>>> 
>>>>> What is that extra power used for?
>>>> 
>>>> Including all sorts of external data sources. Also the logic in the
>>>> lua profiles is not just replaceable by simple key-value pairs, OSM
>>>> requires you to handle a lot of special cases.
>>>> 
>>>>> Presumably I could do the same for world preparation & routing? Have,
>>>>> perhaps a 100GB+ swap file, ideally on an SSD.
>>>> 
>>>> This will fall apart when you have some actual load pressure on the
>>>> system. We need random access to memory, which will create a lot of
>>>> page faults (== slow). Even an SSD is not even close to memory speed.
>>>> 
>>>> You have two options:
>>>> - split the datasets
>>>> - get a bigger server
>>>> 
>>>> Cheers,
>>>> Patrick
>>>> 
>>>> 
>>>> On Thu, Sep 17, 2015 at 10:06 PM, Richard Marsden <winwaed at gmail.com>
>>>> wrote:
>>>>> I've been evaluating OSRM, using it primarily as a library from C++.
>>>>> 
>>>>> I believe I've determined the answer to most of the questions, but I'm
>>>>> also looking for confirmation.
>>>>> (I understand the reason for these constraints - the trade-off of
>>>>> speed vs flexibility)
>>>>> 
>>>>> First, road speeds are set with 'profile.lua' at the osrm-extract
>>>>> stage. This filters out unnecessary roads (eg. foot paths for car
>>>>> routing), but also applies the road speeds.
>>>>> If I wish to change the speed profile, I need to regenerate the road
>>>>> network with osrm-extract and osrm-routed.
>>>>> Correct?
>>>>> 
>>>>> If I wanted different speeds for the final distance/time calculations,
>>>>> I could use the returned route, and apply my own speed table according
>>>>> to the road type of each road segment. This would not, of course,
>>>>> change the route geometry is calculated.
>>>>> 
>>>>> If I want a shortest route (distance optimized) instead of a quickest
>>>>> route (time optimized), I need to set all the road speeds to the same
>>>>> speed and regenerate the network. I.e. osrm does not directly support
>>>>> the concept of a "shortest route".
>>>>> 
>>>>> The profile is provided with a LUA file. I had to look this one up :-)
>>>>> Looks a useful scripting language, but why is this profile a script
>>>>> file, and not a simple configuration file of constants (eg. key-value
>>>>> pairs)?
>>>>> Seems like an unnecessary complexity - I'd like to understand the
>>>>> perceived advantages. What is that extra power used for?
>>>>> 
>>>>> Finally, the memory usage... I saw a reference to the server requiring
>>>>> 40GB of memory for pan-European routing. Presumably that could be
>>>>> offset with a large swap file(?)
>>>>> A large swap file has worked well when I was testing the US-South
>>>>> region on an 8GB machine.
>>>>> Presumably I could do the same for world preparation & routing? Have,
>>>>> perhaps a 100GB+ swap file, ideally on an SSD.
>>>>> 
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Richard Marsden
>>>>> 
>>>>> _______________________________________________
>>>>> OSRM-talk mailing list
>>>>> OSRM-talk at openstreetmap.org
>>>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>>> 
>>>> _______________________________________________
>>>> OSRM-talk mailing list
>>>> OSRM-talk at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>> 
>>> _______________________________________________
>>> OSRM-talk mailing list
>>> OSRM-talk at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/osrm-talk
>> 
>> 
>> 
>> _______________________________________________
>> OSRM-talk mailing list
>> OSRM-talk at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>> 
> 
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk