[OSRM-talk] Required RAM for planet

[Kieran Caplice]

Thanks Daniel and Bjorn,

Will try the increasing the swap file size.

Kind regards,
Kieran Caplice

On 08/03/16 09:51, Daniel Hofmann wrote:
> ||
> |1/ Provide a swapfile (100GB just in case): > fallocate -l 100G 
> /path/to/swapfile > chmod 600 /path/to/swapfile > mkswap 
> /path/to/swapfile > swapon /path/to/swapfile |
>
> Note: this does not write 100 GB of zeros. Instead what it does is 
> allocating a certain amount of blocks and just setting the 
> 'uninitialized' flag on them, returning more or less immediately.
>
>
> 2/ Provide a stxxl config file (250GB should be more than enough, demo 
> server runs this):
>
> > disk=/path/to/stxxl,250000,syscall
>
> See: http://stxxl.sourceforge.net/tags/master/install_config.html
>
>
> Here is some additional explanation: 
> https://github.com/Project-OSRM/osrm-backend/issues/1698#issuecomment-142744361
>
>
> Cheers,
>
> Daniel J H
>
>
> On Tue, Mar 8, 2016 at 10:02 AM, Kieran Caplice 
> <kieran.caplice at temetra.com <mailto:kieran.caplice at temetra.com>> wrote:
>
>     Hi all,
>
>     Just enquiring if anyone has an up-to-date value for the required
>     RAM to extract/prepare the planet PBF on OSRM v4.9.1? Below is the
>     output from extraction on a 64 GB RAM machine:
>
>     [info] Input file: planet-latest.osm.pbf
>     [info] Profile: profile.lua
>     [info] Threads: 12
>     [info] Using script profile.lua
>     [STXXL-MSG] STXXL v1.3.1 (release)
>     [STXXL-MSG] 1 disks are allocated, total space: 250000 MiB
>     [info] Parsing in progress..
>     [info] input file generated by planet-dump-ng 1.1.3
>     [info] timestamp: 2016-02-29T01:59:57Z
>     [info] Using turn restrictions
>     [info] Found 3 exceptions to turn restrictions:
>     [info]   motorcar
>     [info]   motor_vehicle
>     [info]   vehicle
>     [info] Parsing finished after 3630.62 seconds
>     [info] Raw input contains 3240515916 nodes, 333181812 ways, and
>     4046532 relations, and 0 unknown entities
>     [extractor] Sorting used nodes        ... ok, after 301.697s
>     [extractor] Erasing duplicate nodes   ... ok, after 262.325s
>     [extractor] Sorting all nodes         ... ok, after 3419.92s
>     [extractor] Building node id map      ... ok, after 1674.42s
>     [extractor] setting number of nodes   ... ok
>     [extractor] Confirming/Writing used nodes     ... ok, after 837.176s
>     [info] Processed 578533637 nodes
>     [extractor] Sorting edges by start    ... ok, after 1919.47s
>     [extractor] Setting start coords      ... ok, after 2657.74s
>     [extractor] Sorting edges by target   ... ok, after 1894.56s
>     [extractor] Computing edge weights    ... ok, after 2911.29s
>     [extractor] Sorting edges by renumbered start ... ok, after 1864.9s
>     [extractor] Writing used edges       ... ok, after 557.274s
>     [extractor] setting number of edges   ... ok
>     [info] Processed 610970822 edges
>     [extractor] Sorting used ways         ... ok, after 89.8886s
>     [extractor] Sorting 491077 restriction. by from... ok, after 0.906043s
>     [extractor] Fixing restriction starts ... ok, after 39.6105s
>     [extractor] Sorting restrictions. by to  ... ok, after 0.734124s
>     [extractor] Fixing restriction ends   ... ok, after 40.6053s
>     [info] usable restrictions: 459264
>     [extractor] writing street name index ... ok, after 3.3452s
>     [info] extraction finished after 23181.9s
>     [info] Generating edge-expanded graph representation
>     [info]  - 459264 restrictions.
>     [info] Importing n = 578533637 nodes
>     [info]  - 157152 bollard nodes, 792260 traffic lights
>     [info]  and 610970822 edges
>     [info] Graph loaded ok and has 610970822 edges
>     [warn] std::bad_alloc
>
>     From my reading, this is caused by running out of RAM. The only
>     files created were:
>
>     -rw-r--r-- 1 root        root         25G Mar  7 18:57
>     planet-latest.osrm
>     -rw-r--r-- 1 root        root        118M Mar 7 19:00
>     planet-latest.osrm.names
>     -rw-r--r-- 1 root        root         15M Mar 7 19:00
>     planet-latest.osrm.restrictions
>     -rw-r--r-- 1 root        root          20 Mar 7 12:33
>     planet-latest.osrm.timestamp
>
>     Which obviously means preparing gives the following:
>
>     [info] Input file: planet-latest.osrm
>     [info] Profile: profile.lua
>     [info] Threads: 12
>     [info] Loading edge-expanded graph representation
>     [info] Opening planet-latest.osrm.ebg
>     [warn] [exception] osrm input file misses magic number. Check or
>     reprocess the file
>
>     Thanks.
>
>     Kind regards,
>     Kieran Caplice
>
>
>     _______________________________________________
>     OSRM-talk mailing list
>     OSRM-talk at openstreetmap.org <mailto:OSRM-talk at openstreetmap.org>
>     https://lists.openstreetmap.org/listinfo/osrm-talk
>
>
>
>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osrm-talk/attachments/20160310/91a83aea/attachment.html>