<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Thanks Daniel and Bjorn,<br>
<br>
Will try the increasing the swap file size.<br>
<div class="moz-signature"><br>
Kind regards,<br>
Kieran Caplice<br>
<br>
</div>
<div class="moz-cite-prefix">On 08/03/16 09:51, Daniel Hofmann
wrote:<br>
</div>
<blockquote
cite="mid:CAG=qy8d83R2CgMgb6LnNumCNoc59sJZmmxtNxzRweB06XZQQ2A@mail.gmail.com"
type="cite">
<div dir="ltr">
<pre><code></code></pre>
<pre><code>1/ Provide a swapfile (100GB just in case):
> fallocate -l 100G /path/to/swapfile
> chmod 600 /path/to/swapfile
> mkswap /path/to/swapfile
> swapon /path/to/swapfile
</code></pre>
<p>Note: this does not write 100 GB of zeros. Instead what it
does is allocating a certain amount of blocks and just setting
the 'uninitialized' flag on them, returning more or less
immediately.</p>
<p><br>
</p>
<p>2/ Provide a stxxl config file (250GB should be more than
enough, demo server runs this):</p>
<p>> <span class=""></span><span class=""><span class=""></span>disk=/path/to/stxxl,250000,syscall</span><span
class=""><span class=""></span></span></p>
<p>See: <a moz-do-not-send="true"
href="http://stxxl.sourceforge.net/tags/master/install_config.html">http://stxxl.sourceforge.net/tags/master/install_config.html</a></p>
<p><br>
</p>
<p>Here is some additional explanation: <a
moz-do-not-send="true"
href="https://github.com/Project-OSRM/osrm-backend/issues/1698#issuecomment-142744361"><a class="moz-txt-link-freetext" href="https://github.com/Project-OSRM/osrm-backend/issues/1698#issuecomment-142744361">https://github.com/Project-OSRM/osrm-backend/issues/1698#issuecomment-142744361</a></a><br>
</p>
<p><br>
</p>
<p>Cheers,</p>
<p>Daniel J H<br>
</p>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Mar 8, 2016 at 10:02 AM, Kieran
Caplice <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:kieran.caplice@temetra.com" target="_blank">kieran.caplice@temetra.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hi all,<br>
<br>
Just enquiring if anyone has an up-to-date value for the
required RAM to extract/prepare the planet PBF on OSRM
v4.9.1? Below is the output from extraction on a 64 GB RAM
machine:<br>
<br>
<tt>[info] Input file: planet-latest.osm.pbf</tt><tt><br>
</tt><tt>[info] Profile: profile.lua</tt><tt><br>
</tt><tt>[info] Threads: 12</tt><tt><br>
</tt><tt>[info] Using script profile.lua</tt><tt><br>
</tt><tt>[STXXL-MSG] STXXL v1.3.1 (release)</tt><tt><br>
</tt><tt>[STXXL-MSG] 1 disks are allocated, total space:
250000 MiB</tt><tt><br>
</tt><tt>[info] Parsing in progress..</tt><tt><br>
</tt><tt>[info] input file generated by planet-dump-ng
1.1.3</tt><tt><br>
</tt><tt>[info] timestamp: 2016-02-29T01:59:57Z</tt><tt><br>
</tt><tt>[info] Using turn restrictions</tt><tt><br>
</tt><tt>[info] Found 3 exceptions to turn restrictions:</tt><tt><br>
</tt><tt>[info] motorcar</tt><tt><br>
</tt><tt>[info] motor_vehicle</tt><tt><br>
</tt><tt>[info] vehicle</tt><tt><br>
</tt><tt>[info] Parsing finished after 3630.62 seconds</tt><tt><br>
</tt><tt>[info] Raw input contains 3240515916 nodes,
333181812 ways, and 4046532 relations, and 0 unknown
entities</tt><tt><br>
</tt><tt>[extractor] Sorting used nodes ... ok,
after 301.697s</tt><tt><br>
</tt><tt>[extractor] Erasing duplicate nodes ... ok,
after 262.325s</tt><tt><br>
</tt><tt>[extractor] Sorting all nodes ... ok,
after 3419.92s</tt><tt><br>
</tt><tt>[extractor] Building node id map ... ok,
after 1674.42s</tt><tt><br>
</tt><tt>[extractor] setting number of nodes ... ok</tt><tt><br>
</tt><tt>[extractor] Confirming/Writing used nodes ...
ok, after 837.176s</tt><tt><br>
</tt><tt>[info] Processed 578533637 nodes</tt><tt><br>
</tt><tt>[extractor] Sorting edges by start ... ok,
after 1919.47s</tt><tt><br>
</tt><tt>[extractor] Setting start coords ... ok,
after 2657.74s</tt><tt><br>
</tt><tt>[extractor] Sorting edges by target ... ok,
after 1894.56s</tt><tt><br>
</tt><tt>[extractor] Computing edge weights ... ok,
after 2911.29s</tt><tt><br>
</tt><tt>[extractor] Sorting edges by renumbered start ...
ok, after 1864.9s</tt><tt><br>
</tt><tt>[extractor] Writing used edges ... ok,
after 557.274s</tt><tt><br>
</tt><tt>[extractor] setting number of edges ... ok</tt><tt><br>
</tt><tt>[info] Processed 610970822 edges</tt><tt><br>
</tt><tt>[extractor] Sorting used ways ... ok,
after 89.8886s</tt><tt><br>
</tt><tt>[extractor] Sorting 491077 restriction. by
from... ok, after 0.906043s</tt><tt><br>
</tt><tt>[extractor] Fixing restriction starts ... ok,
after 39.6105s</tt><tt><br>
</tt><tt>[extractor] Sorting restrictions. by to ... ok,
after 0.734124s</tt><tt><br>
</tt><tt>[extractor] Fixing restriction ends ... ok,
after 40.6053s</tt><tt><br>
</tt><tt>[info] usable restrictions: 459264</tt><tt><br>
</tt><tt>[extractor] writing street name index ... ok,
after 3.3452s</tt><tt><br>
</tt><tt>[info] extraction finished after 23181.9s</tt><tt><br>
</tt><tt>[info] Generating edge-expanded graph
representation</tt><tt><br>
</tt><tt>[info] - 459264 restrictions.</tt><tt><br>
</tt><tt>[info] Importing n = 578533637 nodes </tt><tt><br>
</tt><tt>[info] - 157152 bollard nodes, 792260 traffic
lights</tt><tt><br>
</tt><tt>[info] and 610970822 edges </tt><tt><br>
</tt><tt>[info] Graph loaded ok and has 610970822 edges</tt><tt><br>
</tt><font color="#990000"><tt>[warn] std::bad_alloc</tt></font><br>
<div><br>
From my reading, this is caused by running out of RAM.
The only files created were:<br>
<br>
<tt>-rw-r--r-- 1 root root 25G Mar 7
18:57 planet-latest.osrm</tt><tt><br>
</tt><tt>-rw-r--r-- 1 root root 118M Mar
7 19:00 planet-latest.osrm.names</tt><tt><br>
</tt><tt>-rw-r--r-- 1 root root 15M Mar
7 19:00 planet-latest.osrm.restrictions</tt><tt><br>
</tt><tt>-rw-r--r-- 1 root root 20 Mar
7 12:33 planet-latest.osrm.timestamp</tt><br>
<br>
Which obviously means preparing gives the following:<br>
<br>
<tt>[info] Input file: planet-latest.osrm</tt><tt><br>
</tt><tt>[info] Profile: profile.lua</tt><tt><br>
</tt><tt>[info] Threads: 12</tt><tt><br>
</tt><tt>[info] Loading edge-expanded graph
representation</tt><tt><br>
</tt><tt>[info] Opening planet-latest.osrm.ebg</tt><tt><br>
</tt><font color="#990000"><tt>[warn] [exception] osrm
input file misses magic number. Check or reprocess
the file</tt></font><br>
<br>
Thanks.<br>
<br>
Kind regards,<br>
Kieran Caplice<br>
<br>
</div>
</div>
<br>
_______________________________________________<br>
OSRM-talk mailing list<br>
<a moz-do-not-send="true"
href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
<a moz-do-not-send="true"
href="https://lists.openstreetmap.org/listinfo/osrm-talk"
rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OSRM-talk mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a>
<a class="moz-txt-link-freetext" href="https://lists.openstreetmap.org/listinfo/osrm-talk">https://lists.openstreetmap.org/listinfo/osrm-talk</a>
</pre>
</blockquote>
<br>
</body>
</html>