[GraphHopper] Running out of heap space when starting webapp with large .osm.pbf

[Nate Atkinson]

Hi Peter,

After installing 64 bit version of Oracle's JRE in place of openjdk and
using the -Xmx7G flag at command line, it looks like things are working.
At least, I'm not getting the same crash.

Now just to wait for GraphHopper to process the 6 GB file. I probably
should have run this on a machine with faster storage.

Thanks for help!

On 10/10/2014 11:20 AM, Peter wrote:
> If you are using the graphhopper.sh script do before:
> export JAVA_OPTS="-Xmx7g -Xms7g"
> 
> Peter
> 
> On 10.10.2014 14:29, Nate Atkinson wrote:
>> Hi,
>>
>> I'm evaluating GraphHopper for integration with some software at work--
>> specifically quickly generating geojson path for routes with up to 40
>> stops. Using a small .osm.pbf file downloaded from geofabrik, it looks
>> like GraphHopper will fit our needs well, and I'm excited to use it.
>>
>> When I try to use a larger .osm.pbf file, like the 6.2 GB file for North
>> America, I get an error when trying to start.
>>
>> Here's what happens:
>>
>> nate at NateTestEnv:~/graphhopper$ java -jar
>> graphhopper-web-0.3-with-dep.jar -Xmx7g jetty.resourcebase=webapp
>> config=config-example.properties
>> osmreader.osm=./north-america-latest.osm.pbf
>>
>> 2014-10-09 13:02:38,259 [main] INFO  com.graphhopper.GraphHopper -
>> version 0.3|2014-05-13T11:36:30+0200 (3,5,3,2,2)
>> 2014-10-09 13:02:38,275 [main] INFO  com.graphhopper.GraphHopper - graph
>> LevelGraphStorage|car|MMAP_STORE_SYNC|2D|,,,,, details:edges:0(0),
>> nodes:0(0), name: - (0), geo:0(0),
>> bounds:1.7976931348623157E308,-1.7976931348623157E308,1.7976931348623157E308,-1.7976931348623157E308
>> 2014-10-09 13:02:38,300 [main] INFO  com.graphhopper.GraphHopper - start
>> creating graph from ./north-america-latest.osm.pbf
>> 2014-10-09 13:02:38,300 [main] INFO  com.graphhopper.GraphHopper - using
>> LevelGraphStorage|car|MMAP_STORE_SYNC|2D|,,,,, memory:totalMB:120, usedMB:8
>> 2014-10-09 13:10:22,788 [main] INFO  graphhopper.reader.OSMReader -
>> 500 000 (preprocess), osmIdMap:6 959 244 (86MB) totalMB:723, usedMB:297
>>
>> ....
>>
>> 2014-10-09 13:13:01,288 [main] INFO  graphhopper.reader.OSMReader -
>> 10 000 000 (preprocess), osmIdMap:137 558 252 (1630MB) totalMB:1895,
>> usedMB:1726
>> Exception in thread "pool-1-thread-2" java.lang.OutOfMemoryError: GC
>> overhead limit exceeded
>> 	at
>> gnu.trove.list.array.TLongArrayList.ensureCapacity(TLongArrayList.java:186)
>> 	at gnu.trove.list.array.TLongArrayList.add(TLongArrayList.java:221)
>> 	at
>> com.graphhopper.reader.pbf.PbfBlobDecoder.processWays(PbfBlobDecoder.java:286)
>> 	at
>> com.graphhopper.reader.pbf.PbfBlobDecoder.processOsmPrimitives(PbfBlobDecoder.java:372)
>> 	at
>> com.graphhopper.reader.pbf.PbfBlobDecoder.runAndTrapExceptions(PbfBlobDecoder.java:388)
>> 	at com.graphhopper.reader.pbf.PbfBlobDecoder.run(PbfBlobDecoder.java:406)
>> ....
>>
>>
>> I get a similar message for a second thread. The process stays running,
>> but I see no CPU activity in top.
>>
>> The system is running Debian wheezy. I tested with openjdk 1.6 and 1.7.
>>
>> So, how can I get this running with a large network? I'm sure there's a
>> command line switch or config setting somewhere that I've missed due to
>> my inexperience with java.
>>
>> Best regards,
>>
>> Nate Atkinson
>>
>> _______________________________________________
>> GraphHopper mailing list
>> GraphHopper at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/graphhopper
> 
> 
> _______________________________________________
> GraphHopper mailing list
> GraphHopper at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/graphhopper
>