[GraphHopper] Out of memory at graph creation

[Nicolas Gillet]

(error sending previous mail)

Hi Peter,

Thanks for your quick reply.
The data are travel time histograms.
For each edge I have two histograms (backward and forward direction).
Each histograms splits the day into 48 slots of 30 minutes. For each of this slot I have an integer value representing the travel time (in second) on the edge at the time slot.
Knowing that I have 854 541 edges and 7 days in a week the estimated size could be 854541 *2 * 48 * 7 = 574 251 552 integers => 2 297 006 208 Bytes
Now that I have computed it, I think I'll have to reduce it maybe encoding the time as a byte instead of an int, but it would only reduce by 4.

The stack trace extract :
java.lang.OutOfMemoryError: Java heap space - problem when allocating new memory. Old capacity: 178257920, new bytes:1048576, segmentSizeIntsPower:20, new segments:1, existing:170
      at com.graphhopper.storage.RAMDataAccess.ensureCapacity(RAMDataAccess.java:135)
      at com.graphhopper.storage.GraphStorage.incCapacity(GraphStorage.java:262)
      at com.graphhopper.storage.traveltime.WeekTTFGraphStorage.ensureTTFIndex(WeekTTFGraphStorage.java:318)
      at com.graphhopper.storage.traveltime.WeekTTFGraphStorage.newTravelTimeFunction(WeekTTFGraphStorage.java:301)
      at com.graphhopper.reader.MultiNetReaderHelper.linkTTFToEdge(MultiNetReaderHelper.java:250)
      at com.graphhopper.reader.MultiNetReader.writeSpeedProfiles(MultiNetReader.java:258)
      at com.graphhopper.reader.MultiNetReader.writeMn2Graph(MultiNetReader.java:142)
      at com.graphhopper.reader.MultiNetReader.mn2Graph(MultiNetReader.java:273)
      at com.graphhopper.reader.MultiNetReader.importGraph(MultiNetReader.java:369)
      at com.graphhopper.GraphHopper.importGraph(GraphHopper.java:571)
      at com.graphhopper.GraphHopper.importGraph(GraphHopper.java:514)
      at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:475)

Which is strange is that I also have the problem using the MMapDataAccess.

I keep investigating
Nicolas GILLET


Market-IP - Creating Mobile Intelligence
Phone : +32 81 33 11 11
Fax : +32 81 33 11 10
www.market-ip.com<http://www.market-ip.com/> - www.telefleet.com<http://www.telefleet.com/> - www.geoplanning.net<http://www.geoplanning.net/> - www.drivexpert.net<http://www.drivexpert.net/>

De : Peter K [mailto:peathal at yahoo.de]
Envoyé : mardi 2 juillet 2013 11:18
À : GraphHopper Java routing engine
Objet : Re: [GraphHopper] Out of memory at graph creation

Hi Nicolas,

creating world wide graph needs lots of RAM (normal graph needs 8GB, CH needs 13GB, creating the latter one needs ~22GB). Although for only belgium territority this should be sufficient. Not sure how your additional data source looks like and what the exact reason of the OOM might cause. What is the exact stacktrace?

Did you increase the default -Xmx1000m settings either by editing graphhopper.sh or by providing your own JAVA_OPTS? (export JAVA_OPTS="-Xmx2000m ..."; ./graphhopper.sh ...)

> I'd like to know if there could be a way to flush the DataAccess on a regular basis to release memory.

You could use the MMapDataAccess but you'll have to make this thread-safe (e.g. with a simple DataAccess wrapper). But if not enough RAM is available => everything will be slower (import + queries)

Regards,
Peter.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20130702/dee273a8/attachment.html>