<div dir="ltr"><pre><code></code></pre><pre><code>1/ Provide a swapfile (100GB just in case):<br><br>> fallocate -l 100G /path/to/swapfile
> chmod 600 /path/to/swapfile
> mkswap /path/to/swapfile
> swapon /path/to/swapfile
</code></pre>

<p>Note: this does not write 100 GB of zeros. Instead what it does is 
allocating a certain amount of blocks and just setting the 
'uninitialized' flag on them, returning more or less immediately.</p><p><br></p><p>2/ Provide a stxxl config file (250GB should be more than enough, demo server runs this):</p><p>> <span class=""></span><span class=""><span class=""></span>disk=/path/to/stxxl,250000,syscall</span><span class=""><span class=""></span></span></p><p>See: <a href="http://stxxl.sourceforge.net/tags/master/install_config.html">http://stxxl.sourceforge.net/tags/master/install_config.html</a></p><p><br></p><p>Here is some additional explanation: <a href="https://github.com/Project-OSRM/osrm-backend/issues/1698#issuecomment-142744361">https://github.com/Project-OSRM/osrm-backend/issues/1698#issuecomment-142744361</a><br></p><p><br></p><p>Cheers,</p><p>Daniel J H<br></p></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 8, 2016 at 10:02 AM, Kieran Caplice <span dir="ltr"><<a href="mailto:kieran.caplice@temetra.com" target="_blank">kieran.caplice@temetra.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  

    
  
  <div text="#000000" bgcolor="#FFFFFF">
    Hi all,<br>
    <br>
    Just enquiring if anyone has an up-to-date value for the required
    RAM to extract/prepare the planet PBF on OSRM v4.9.1? Below is the
    output from extraction on a 64 GB RAM machine:<br>
    <br>
    <tt>[info] Input file: planet-latest.osm.pbf</tt><tt><br>
    </tt><tt>[info] Profile: profile.lua</tt><tt><br>
    </tt><tt>[info] Threads: 12</tt><tt><br>
    </tt><tt>[info] Using script profile.lua</tt><tt><br>
    </tt><tt>[STXXL-MSG] STXXL v1.3.1 (release)</tt><tt><br>
    </tt><tt>[STXXL-MSG] 1 disks are allocated, total space: 250000 MiB</tt><tt><br>
    </tt><tt>[info] Parsing in progress..</tt><tt><br>
    </tt><tt>[info] input file generated by planet-dump-ng 1.1.3</tt><tt><br>
    </tt><tt>[info] timestamp: 2016-02-29T01:59:57Z</tt><tt><br>
    </tt><tt>[info] Using turn restrictions</tt><tt><br>
    </tt><tt>[info] Found 3 exceptions to turn restrictions:</tt><tt><br>
    </tt><tt>[info]   motorcar</tt><tt><br>
    </tt><tt>[info]   motor_vehicle</tt><tt><br>
    </tt><tt>[info]   vehicle</tt><tt><br>
    </tt><tt>[info] Parsing finished after 3630.62 seconds</tt><tt><br>
    </tt><tt>[info] Raw input contains 3240515916 nodes, 333181812 ways,
      and 4046532 relations, and 0 unknown entities</tt><tt><br>
    </tt><tt>[extractor] Sorting used nodes        ... ok, after
      301.697s</tt><tt><br>
    </tt><tt>[extractor] Erasing duplicate nodes   ... ok, after
      262.325s</tt><tt><br>
    </tt><tt>[extractor] Sorting all nodes         ... ok, after
      3419.92s</tt><tt><br>
    </tt><tt>[extractor] Building node id map      ... ok, after
      1674.42s</tt><tt><br>
    </tt><tt>[extractor] setting number of nodes   ... ok</tt><tt><br>
    </tt><tt>[extractor] Confirming/Writing used nodes     ... ok, after
      837.176s</tt><tt><br>
    </tt><tt>[info] Processed 578533637 nodes</tt><tt><br>
    </tt><tt>[extractor] Sorting edges by start    ... ok, after
      1919.47s</tt><tt><br>
    </tt><tt>[extractor] Setting start coords      ... ok, after
      2657.74s</tt><tt><br>
    </tt><tt>[extractor] Sorting edges by target   ... ok, after
      1894.56s</tt><tt><br>
    </tt><tt>[extractor] Computing edge weights    ... ok, after
      2911.29s</tt><tt><br>
    </tt><tt>[extractor] Sorting edges by renumbered start ... ok, after
      1864.9s</tt><tt><br>
    </tt><tt>[extractor] Writing used edges       ... ok, after 557.274s</tt><tt><br>
    </tt><tt>[extractor] setting number of edges   ... ok</tt><tt><br>
    </tt><tt>[info] Processed 610970822 edges</tt><tt><br>
    </tt><tt>[extractor] Sorting used ways         ... ok, after
      89.8886s</tt><tt><br>
    </tt><tt>[extractor] Sorting 491077 restriction. by from... ok,
      after 0.906043s</tt><tt><br>
    </tt><tt>[extractor] Fixing restriction starts ... ok, after
      39.6105s</tt><tt><br>
    </tt><tt>[extractor] Sorting restrictions. by to  ... ok, after
      0.734124s</tt><tt><br>
    </tt><tt>[extractor] Fixing restriction ends   ... ok, after
      40.6053s</tt><tt><br>
    </tt><tt>[info] usable restrictions: 459264</tt><tt><br>
    </tt><tt>[extractor] writing street name index ... ok, after 3.3452s</tt><tt><br>
    </tt><tt>[info] extraction finished after 23181.9s</tt><tt><br>
    </tt><tt>[info] Generating edge-expanded graph representation</tt><tt><br>
    </tt><tt>[info]  - 459264 restrictions.</tt><tt><br>
    </tt><tt>[info] Importing n = 578533637 nodes </tt><tt><br>
    </tt><tt>[info]  - 157152 bollard nodes, 792260 traffic lights</tt><tt><br>
    </tt><tt>[info]  and 610970822 edges </tt><tt><br>
    </tt><tt>[info] Graph loaded ok and has 610970822 edges</tt><tt><br>
    </tt><font color="#990000"><tt>[warn] std::bad_alloc</tt></font><br>
    <div><br>
      From my reading, this is caused by running out of RAM. The only
      files created were:<br>
      <br>
      <tt>-rw-r--r-- 1 root        root         25G Mar  7 18:57
        planet-latest.osrm</tt><tt><br>
      </tt><tt>-rw-r--r-- 1 root        root        118M Mar  7 19:00
        planet-latest.osrm.names</tt><tt><br>
      </tt><tt>-rw-r--r-- 1 root        root         15M Mar  7 19:00
        planet-latest.osrm.restrictions</tt><tt><br>
      </tt><tt>-rw-r--r-- 1 root        root          20 Mar  7 12:33
        planet-latest.osrm.timestamp</tt><br>
      <br>
      Which obviously means preparing gives the following:<br>
      <br>
      <tt>[info] Input file: planet-latest.osrm</tt><tt><br>
      </tt><tt>[info] Profile: profile.lua</tt><tt><br>
      </tt><tt>[info] Threads: 12</tt><tt><br>
      </tt><tt>[info] Loading edge-expanded graph representation</tt><tt><br>
      </tt><tt>[info] Opening planet-latest.osrm.ebg</tt><tt><br>
      </tt><font color="#990000"><tt>[warn] [exception] osrm input file
          misses magic number. Check or reprocess the file</tt></font><br>
      <br>
      Thanks.<br>
      <br>
      Kind regards,<br>
      Kieran Caplice<br>
      <br>
    </div>
  </div>

<br>_______________________________________________<br>
OSRM-talk mailing list<br>
<a href="mailto:OSRM-talk@openstreetmap.org">OSRM-talk@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/osrm-talk" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/osrm-talk</a><br>
<br></blockquote></div><br></div>