<div dir="ltr">Hi Kieran,<div><br></div><div> We don't, but I've done some profiling in the past, Docker itself should have negligible impact.</div><div><br></div><div> The two things that might be slowing things down:</div><div><br></div><div> 1) The docker images don't bundle jemalloc (<a href="http://jemalloc.net/">http://jemalloc.net/</a>), which can speed things up by 10-15%</div><div> 2) osrm-contract is very CPU intensive - the more threads you give it, the faster it will run. Give it more if you can.</div><div><br></div><div> 256GB of RAM should be more than enough to avoid swapping, but you might want to disable swap to be sure - if it's being used, it will have a huge impact on speed.</div><div><br></div><div>daniel</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 22, 2017 at 2:30 AM, Kieran Caplice <span dir="ltr"><<a href="mailto:kieran.caplice@temetra.com" target="_blank">kieran.caplice@temetra.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Daniel,</p>
<div class="m_-3505390796180504946moz-signature">Can you clarify if you use Docker at
Mapbox? What kind of server do you guys have (if not the one
described on the wiki page)?<br>
<br>
Over night, the process has gone to just 75% complete from 65%
yesterday - 41 hours in total now, and back to maxing CPU again.<br>
<br>
Kind regards,<br>
Kieran Caplice<br>
<br>
</div><div><div class="h5">
<div class="m_-3505390796180504946moz-cite-prefix">On 21/09/17 17:17, Daniel Patterson
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">OSRM supports *two* core routing algorithms - CH
and MLD. The `osrm-contract` tool generates the CH dataset, but
you can use the MLD pipeline instead with:
<div><br>
</div>
<div>osrm-extract -p profiles/bicycle.lua yourmap.osm.pbf</div>
<div>osrm-partition yourmap.osrm</div>
<div>osrm-customize yourmap.osrm</div>
<div>osrm-routed -a MLD yourmap.osrm</div>
<div><br>
</div>
<div>This sequence of tools should be significantly quicker than
osrm-contract - the price you pay is that routing requests are
about 5x slower (still pretty fast though!). The reason that
MLD exists is for the `osrm-customize` step - it allows you to
import traffic data very quickly and update the routing graph
(~1 minute for North America).</div>
<div><br>
</div>
<div>It's hard to say exactly what's going wrong with
osrm-contract here - here at Mapbox, we daily run
`osrm-contract` over the latest planet with the bicycle
profile without a problem, however, Alex and others have
reported issues with what seem like hangs on much smaller
datasets that we've been unable to reproduce so far.</div>
<div><br>
</div>
<div>The runtime of osrm-contract is affected by how much
hierarchy exists in the data - the more similar the edge
speeds (like in foot) and the more edges there are, the slower
it gets, often in a non-linear fashion. The car profile has a
very hierarchical structure (many different road speeds), so
it fits well into the CH, the construction algorithm doesn't
need to compare as many options.</div>
<div><br>
</div>
<div>daniel</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Sep 21, 2017 at 9:03 AM, Kieran
Caplice <span dir="ltr"><<a href="mailto:kieran.caplice@temetra.com" target="_blank">kieran.caplice@temetra.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>We're actually looking for the best of both car and
foot, so in my head, bicycle would be the happy medium
(though I could be completely wrong on this).<br>
</p>
<div class="m_-3505390796180504946m_6646586754541093830moz-signature"> Kind
regards,<br>
Kieran Caplice<br>
<br>
</div>
<div>
<div class="m_-3505390796180504946h5">
<div class="m_-3505390796180504946m_6646586754541093830moz-cite-prefix">On
21/09/17 16:53, Alex Farioletti wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">i've run into the same issues, and
now i just use metroextracts of the areas that i
need for the bike stuff i do and it reduces the
time significantly</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_-3505390796180504946m_6646586754541093830gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr"><b>Alex Farioletti</b><br>
<div><b><a value="+14153121674">415.312.1674</a></b>
<div><i style="font-weight:bold"><a href="http://tcbcourier.com" target="_blank">tcbcourier.com</a> </i></div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Thu, Sep 21, 2017 at
8:49 AM, Kieran Caplice <span dir="ltr"><<a href="mailto:kieran.caplice@temetra.com" target="_blank">kieran.caplice@temetra.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Thanks Daniel.</p>
<p>I'm using the bicycle profile, so I would
expect based on what you've said that
somewhere up to 36 hours would be likely.
However, this is the current output, after
25h40m:</p>
<div class="m_-3505390796180504946m_6646586754541093830m_-3407091792784772007moz-signature"><tt>[info]
Input file:
/data/1505492056/planet-latest<wbr>.osrm</tt><tt><br>
</tt><tt>[info] Threads: 12</tt><tt><br>
</tt><tt>[info] Reading node weights.</tt><tt><br>
</tt><tt>[info] Done reading node weights.</tt><tt><br>
</tt><tt>[info] Loading edge-expanded
graph representation</tt><tt><br>
</tt><tt>[info] merged 2379332 edges out
of 1777752432</tt><tt><br>
</tt><tt>[info] initializing node
priorities... ok.</tt><tt><br>
</tt><tt>[info] preprocessing 389797971
(90%) nodes...</tt><tt><br>
</tt><tt>[info] . 10% . 20% . 30% . 40% .
50% . 60% </tt><br>
<br>
It hasn't advanced past 60% in the last
2-3 hours. It is however maxing CPU and
using approximately the same amount of RAM
since it started.<br>
<br>
Kind regards,<br>
Kieran Caplice<br>
<br>
</div>
<div>
<div class="m_-3505390796180504946m_6646586754541093830h5">
<div class="m_-3505390796180504946m_6646586754541093830m_-3407091792784772007moz-cite-prefix">On
21/09/17 16:39, Daniel Patterson
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Kieran,
<div><br>
</div>
<div> The contraction time will be
slow - many, many hours for the
whole planet. *Typically* for the
car profile it's about 12 hours,
but if you use bike or foot, or
your own profile, it can get a lot
bigger.</div>
<div><br>
</div>
<div> If you've messed with the
travel speeds, that can have a big
effect too. 24 hours is not
unheard of, but whether it's legit
will depend a lot on the details.</div>
<div><br>
</div>
<div>daniel</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Sep
21, 2017 at 7:00 AM, Kieran
Caplice <span dir="ltr"><<a href="mailto:kieran.caplice@temetra.com" target="_blank">kieran.caplice@temetra.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
Could anyone give an approx
estimate for the time required
to run the osrm-contract on
planet data on a 12 thread, 256
GB RAM, SSD machine?<br>
<br>
The osrm-extract process
finished in 232 minutes, but the
contract has now been running
solid for 24 hours, and appears
to be stuck at 60% on
"preprocessing nodes". All 12
cores are generally maxed out,
and the process is using nearly
90 GB of RAM.<br>
<br>
This is the second time I've run
the contract process, as my SSH
connection to the server dropped
the first time and the process
wasn't running in a screen etc,
so I assumed after the 40-odd
hours it was running for, the
connection drop caused it to
hang, but now I'm not so sure.
Were there any files I should
maybe have cleared before trying
to run it again?<br>
<br>
I'm using the docker image to
run the command (using
osrm/osrm-backend:latest): time
docker run -t -v
/opt/osrm/data:/data
osrm/osrm-backend osrm-contract
/data/1505492056/planet-latest<wbr>.osrm<br>
<br>
Kind regards,<br>
Kieran Caplice<br>
<br>
<br>
______________________________<wbr>_________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.or<wbr>g/listinfo/osrm-talk</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_-3505390796180504946m_6646586754541093830m_-3407091792784772007mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
OSRM-talk mailing list
<a class="m_-3505390796180504946m_6646586754541093830m_-3407091792784772007moz-txt-link-abbreviated" href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a>
<a class="m_-3505390796180504946m_6646586754541093830m_-3407091792784772007moz-txt-link-freetext" href="https://lists.openstreetmap.org/listinfo/osrm-talk" target="_blank">https://lists.openstreetmap.or<wbr>g/listinfo/osrm-talk</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.or<wbr>g/listinfo/osrm-talk</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_-3505390796180504946m_6646586754541093830mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
OSRM-talk mailing list
<a class="m_-3505390796180504946m_6646586754541093830moz-txt-link-abbreviated" href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a>
<a class="m_-3505390796180504946m_6646586754541093830moz-txt-link-freetext" href="https://lists.openstreetmap.org/listinfo/osrm-talk" target="_blank">https://lists.openstreetmap.or<wbr>g/listinfo/osrm-talk</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.or<wbr>g/listinfo/osrm-talk</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_-3505390796180504946mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
OSRM-talk mailing list
<a class="m_-3505390796180504946moz-txt-link-abbreviated" href="mailto:OSRM-talk@openstreetmap.org" target="_blank">OSRM-talk@openstreetmap.org</a>
<a class="m_-3505390796180504946moz-txt-link-freetext" href="https://lists.openstreetmap.org/listinfo/osrm-talk" target="_blank">https://lists.openstreetmap.<wbr>org/listinfo/osrm-talk</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>______________________________<wbr>_________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.<wbr>org/listinfo/osrm-talk</a><br>
<br></blockquote></div><br></div>