[GraphHopper] Out of memory at graph creation

[Peter K]

Hi Nicolas,

that kind of information should be probably stored in a MMapDataAccess
as you will often request the current time (I guess) and so the OS does
not need to put this data away on disc.
(+do not forget the sync wrapper)

Or you could manage this on your own in a new DataAccess class and
additionally try to compress that data and maintain a kind of a cache.
see some compression code in CompressedArray

> Which is strange is that I also have the problem using the MMapDataAccess.


Hmmh, are you on a 32bit machine? And are you making sure the JVM gets
enough RAM (via -Xmx)?

Regards,
Peter.

> Hi Peter,
>
>  
>
> Thanks for your quick reply.
>
> The data are travel time histograms.
>
> For each edge I have two histograms (backward and forward direction).
>
> Each histograms splits the day into 48 slots of 30 minutes. For each
> of this slot I have an integer value representing the travel time (in
> second) on the edge at the time slot.
>
> Knowing that I have 854 541 edges and 7 days in a week the estimated
> size could be 854541 *2 * 48 * 7 = 574 251 552 integers => 2 297 006
> 208 Bytes
>
> Now that I have computed it, I think I'll have to reduce it maybe
> encoding the time as a byte instead of an int, but it would only
> reduce by 4.
>
>  
>
> The stack trace extract :
>
>  
>
> Nicolas GILLET
>
>  
>
> * *
>
> *Market-IP --*///Creating Mobile Intelligence/
>
> Phone : +32 81 33 11 11
>
> Fax : +32 81 33 11 10
>
> www.market-ip.com <http://www.market-ip.com/> -- www.telefleet.com
> <http://www.telefleet.com/> -- www.geoplanning.net
> <http://www.geoplanning.net/> -- www.drivexpert.net
> <http://www.drivexpert.net/>
>
> Description : cid:image003.png at 01CD5521.849F3340
> <http://www.linkedin.com/groups/MarketIP-4289716?gid=4289716&trk=hb_side_g>
>
>  
>
> *De :*Peter K [mailto:peathal at yahoo.de]
> *Envoyé :* mardi 2 juillet 2013 11:18
> *À :* GraphHopper Java routing engine
> *Objet :* Re: [GraphHopper] Out of memory at graph creation
>
>  
>
> Hi Nicolas,
>
> creating world wide graph needs lots of RAM (normal graph needs 8GB,
> CH needs 13GB, creating the latter one needs ~22GB). Although for only
> belgium territority this should be sufficient. Not sure how your
> additional data source looks like and what the exact reason of the OOM
> might cause. What is the exact stacktrace?
>
> Did you increase the default -Xmx1000m settings either by editing
> graphhopper.sh or by providing your own JAVA_OPTS? (export
> JAVA_OPTS="-Xmx2000m ..."; ./graphhopper.sh ...)
>
> > I'd like to know if there could be a way to flush the DataAccess on a regular basis to release memory.
>
> You could use the MMapDataAccess but you'll have to make this
> thread-safe (e.g. with a simple DataAccess wrapper). But if not enough
> RAM is available => everything will be slower (import + queries)
>
> Regards,
> Peter.
>
>     Hello,
>
>      
>
>     I'm facing an OutOfMemoryError (heap space) when creating a bigger
>     graph.
>
>     Actually, I am creating a graph out of our own data source
>     including speed profile definitions on the Belgian territory.
>
>     Creating a simple graph works fine but when it comes to create the
>     speed profiles, my process quickly runs out of memory.
>
>     To store these speed profiles, I have created 7 more DataAccess,
>     one for each day of week and each edge has one speed profile
>     stored in there.
>
>     I'd like to know if there could be a way to flush the DataAccess
>     on a regular basis to release memory.
>
>      
>
>     When I see that you managed to create a worldwide graph, I guess
>     there might be such mechanism but I couldn't find it.
>
>      
>
>     Thank you
>
>      
>
>     Nicolas GILLET
>
>      
>
>     * *
>
>     *Market-IP --*///Creating Mobile Intelligence/
>
>     Phone : +32 81 33 11 11
>
>     Fax : +32 81 33 11 10
>
>     www.market-ip.com <http://www.market-ip.com/> -- www.telefleet.com
>     <http://www.telefleet.com/> -- www.geoplanning.net
>     <http://www.geoplanning.net/> -- www.drivexpert.net
>     <http://www.drivexpert.net/>
>
>      
>
>
>
>
>     _______________________________________________
>
>     GraphHopper mailing list
>
>     GraphHopper at openstreetmap.org <mailto:GraphHopper at openstreetmap.org>
>
>     http://lists.openstreetmap.org/listinfo/graphhopper
>
>  
>
>
>
> _______________________________________________
> GraphHopper mailing list
> GraphHopper at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/graphhopper

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20130702/68b29107/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 1263 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20130702/68b29107/attachment.png>