<div dir="ltr"><div><div>If osrm-prepare crashed, just add more swap. Also make sure to read this ticket:<br>> <a href="https://github.com/Project-OSRM/osrm-backend/issues/1698">https://github.com/Project-OSRM/osrm-backend/issues/1698</a><br><br></div><div>I don't think we have documentation for the profiles. I know they are somewhat ugly and huge. I would recommend taking a look at the profile's general structure first, so that you roughly understand e.g. that there are callbacks for OSM nodes and ways, and what they do. After that I would simply trace the globals through the script. That is e.g. search for all places where a certain whitelist is used and build a mental model based on that. Pen and paper may help :)<br></div><div><br></div>> <a href="https://github.com/Project-OSRM/osrm-backend/blob/develop/profiles/car.lua">https://github.com/Project-OSRM/osrm-backend/blob/develop/profiles/car.lua</a><br><br></div>You can find the C++ side of things in the extractor directory:<br><div><div><div><div>> <a href="https://github.com/Project-OSRM/osrm-backend/tree/develop/extractor">https://github.com/Project-OSRM/osrm-backend/tree/develop/extractor</a><br><br></div><div>For example, the scripting environment uses luabind to make a few functions visible, e.g. the get_value_by_key function, that then can be used in the profiles. The other way, that is using lua functions in C++ is mostly done in the extractor implementation: I would recommend just searching for the symbols you want to know more about:<br></div><div><br>> <a href="https://github.com/Project-OSRM/osrm-backend/search?utf8=%E2%9C%93&q=node_function&type=Code">https://github.com/Project-OSRM/osrm-backend/search?utf8=%E2%9C%93&q=node_function&type=Code</a><br><br></div><div>Cheers,<br></div><div>Daniel<br></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 12, 2015 at 4:00 AM, Richard Marsden <span dir="ltr"><<a href="mailto:winwaed@gmail.com" target="_blank">winwaed@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It is possible I've just managed to demonstrate this to myself.<br>
osrm-prepare crashed out on the Europe-wide dataset at the 90+%<br>
complete level with 30GB swap (16GB real). For much of the run it<br>
didn't go above about 15GB swap. Very slow of course - but that was to<br>
be expected.<br>
<br>
This continues to be a side / evaluation project, but I'm continuing<br>
to investigate. Looks like the PC in question should be able to handle<br>
most countries, but will balk at most continents, and the very largest<br>
countries (e.g. the US). High-RAM hardware is relatively pricey, but<br>
it is a possibility in the new year.<br>
<br>
Looking at the individual pieces, next I'm going to look at the<br>
profile.lua file. I've never used LUA before, but that is easy enough<br>
to remedy with online tutorials and the various books that are<br>
available. But what about the script that is used by OSRM? Is there<br>
any documentation as to the variables and/or functions that are<br>
expected? And the tags used in OSM that the script is expected to<br>
map?<br>
<br>
<br>
Cheers,<br>
<br>
Richard Marsden<br>
<span class="im HOEnZb"><br>
On Thu, Sep 17, 2015 at 4:37 PM, Patrick Niklaus<br>
<<a href="mailto:patrick.niklaus@student.kit.edu">patrick.niklaus@student.kit.edu</a>> wrote:<br>
</span><div class="HOEnZb"><div class="h5">> W.r.t. the pre-preprocessing you are correct.<br>
><br>
>> What is that extra power used for?<br>
><br>
> Including all sorts of external data sources. Also the logic in the<br>
> lua profiles is not just replaceable by simple key-value pairs, OSM<br>
> requires you to handle a lot of special cases.<br>
><br>
>> Presumably I could do the same for world preparation & routing? Have, perhaps a 100GB+ swap file, ideally on an SSD.<br>
><br>
> This will fall apart when you have some actual load pressure on the<br>
> system. We need random access to memory, which will create a lot of<br>
> page faults (== slow). Even an SSD is not even close to memory speed.<br>
><br>
> You have two options:<br>
> - split the datasets<br>
> - get a bigger server<br>
><br>
> Cheers,<br>
> Patrick<br>
><br>
><br>
> On Thu, Sep 17, 2015 at 10:06 PM, Richard Marsden <<a href="mailto:winwaed@gmail.com">winwaed@gmail.com</a>> wrote:<br>
>> I've been evaluating OSRM, using it primarily as a library from C++.<br>
>><br>
>> I believe I've determined the answer to most of the questions, but I'm<br>
>> also looking for confirmation.<br>
>> (I understand the reason for these constraints - the trade-off of<br>
>> speed vs flexibility)<br>
>><br>
>> First, road speeds are set with 'profile.lua' at the osrm-extract<br>
>> stage. This filters out unnecessary roads (eg. foot paths for car<br>
>> routing), but also applies the road speeds.<br>
>> If I wish to change the speed profile, I need to regenerate the road<br>
>> network with osrm-extract and osrm-routed.<br>
>> Correct?<br>
>><br>
>> If I wanted different speeds for the final distance/time calculations,<br>
>> I could use the returned route, and apply my own speed table according<br>
>> to the road type of each road segment. This would not, of course,<br>
>> change the route geometry is calculated.<br>
>><br>
>> If I want a shortest route (distance optimized) instead of a quickest<br>
>> route (time optimized), I need to set all the road speeds to the same<br>
>> speed and regenerate the network. I.e. osrm does not directly support<br>
>> the concept of a "shortest route".<br>
>><br>
>> The profile is provided with a LUA file. I had to look this one up :-)<br>
>> Looks a useful scripting language, but why is this profile a script<br>
>> file, and not a simple configuration file of constants (eg. key-value<br>
>> pairs)?<br>
>> Seems like an unnecessary complexity - I'd like to understand the<br>
>> perceived advantages. What is that extra power used for?<br>
>><br>
>> Finally, the memory usage... I saw a reference to the server requiring<br>
>> 40GB of memory for pan-European routing. Presumably that could be<br>
>> offset with a large swap file(?)<br>
>> A large swap file has worked well when I was testing the US-South<br>
>> region on an 8GB machine.<br>
>> Presumably I could do the same for world preparation & routing? Have,<br>
>> perhaps a 100GB+ swap file, ideally on an SSD.<br>
>><br>
>><br>
>> Cheers,<br>
>><br>
>> Richard Marsden<br>
>><br>
>> _______________________________________________<br>
>> OSRM-talk mailing list<br>
>> <a href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
>> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
><br>
> _______________________________________________<br>
> OSRM-talk mailing list<br>
> <a href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
> <a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
<br>
_______________________________________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
</div></div></blockquote></div><br></div>