[GraphHopper] Out of memory at graph creation

[Nicolas Gillet]

Hi Peter,

Thanks for your quick reply.
The data are travel time histograms.
For each edge I have two histograms (backward and forward direction).
Each histograms splits the day into 48 slots of 30 minutes. For each of this slot I have an integer value representing the travel time (in second) on the edge at the time slot.
Knowing that I have 854 541 edges and 7 days in a week the estimated size could be 854541 *2 * 48 * 7 = 574 251 552 integers => 2 297 006 208 Bytes
Now that I have computed it, I think I'll have to reduce it maybe encoding the time as a byte instead of an int, but it would only reduce by 4.

The stack trace extract :

Nicolas GILLET


Market-IP - Creating Mobile Intelligence
Phone : +32 81 33 11 11
Fax : +32 81 33 11 10
www.market-ip.com<http://www.market-ip.com/> - www.telefleet.com<http://www.telefleet.com/> - www.geoplanning.net<http://www.geoplanning.net/> - www.drivexpert.net<http://www.drivexpert.net/>
[Description : cid:image003.png at 01CD5521.849F3340]<http://www.linkedin.com/groups/MarketIP-4289716?gid=4289716&trk=hb_side_g>

De : Peter K [mailto:peathal at yahoo.de]
Envoyé : mardi 2 juillet 2013 11:18
À : GraphHopper Java routing engine
Objet : Re: [GraphHopper] Out of memory at graph creation

Hi Nicolas,

creating world wide graph needs lots of RAM (normal graph needs 8GB, CH needs 13GB, creating the latter one needs ~22GB). Although for only belgium territority this should be sufficient. Not sure how your additional data source looks like and what the exact reason of the OOM might cause. What is the exact stacktrace?

Did you increase the default -Xmx1000m settings either by editing graphhopper.sh or by providing your own JAVA_OPTS? (export JAVA_OPTS="-Xmx2000m ..."; ./graphhopper.sh ...)

> I'd like to know if there could be a way to flush the DataAccess on a regular basis to release memory.

You could use the MMapDataAccess but you'll have to make this thread-safe (e.g. with a simple DataAccess wrapper). But if not enough RAM is available => everything will be slower (import + queries)

Regards,
Peter.
Hello,

I'm facing an OutOfMemoryError (heap space) when creating a bigger graph.
Actually, I am creating a graph out of our own data source including speed profile definitions on the Belgian territory.
Creating a simple graph works fine but when it comes to create the speed profiles, my process quickly runs out of memory.
To store these speed profiles, I have created 7 more DataAccess, one for each day of week and each edge has one speed profile stored in there.
I'd like to know if there could be a way to flush the DataAccess on a regular basis to release memory.

When I see that you managed to create a worldwide graph, I guess there might be such mechanism but I couldn't find it.

Thank you

Nicolas GILLET


Market-IP - Creating Mobile Intelligence
Phone : +32 81 33 11 11
Fax : +32 81 33 11 10
www.market-ip.com<http://www.market-ip.com/> - www.telefleet.com<http://www.telefleet.com/> - www.geoplanning.net<http://www.geoplanning.net/> - www.drivexpert.net<http://www.drivexpert.net/>





_______________________________________________

GraphHopper mailing list

GraphHopper at openstreetmap.org<mailto:GraphHopper at openstreetmap.org>

http://lists.openstreetmap.org/listinfo/graphhopper

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20130702/8d91dbc9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1263 bytes
Desc: image001.png
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20130702/8d91dbc9/attachment.png>