[GraphHopper] Running out of heap space when starting webapp with large .osm.pbf

[Nate Atkinson]

Hi,

I'm evaluating GraphHopper for integration with some software at work--
specifically quickly generating geojson path for routes with up to 40
stops. Using a small .osm.pbf file downloaded from geofabrik, it looks
like GraphHopper will fit our needs well, and I'm excited to use it.

When I try to use a larger .osm.pbf file, like the 6.2 GB file for North
America, I get an error when trying to start.

Here's what happens:

nate at NateTestEnv:~/graphhopper$ java -jar
graphhopper-web-0.3-with-dep.jar -Xmx7g jetty.resourcebase=webapp
config=config-example.properties
osmreader.osm=./north-america-latest.osm.pbf

2014-10-09 13:02:38,259 [main] INFO  com.graphhopper.GraphHopper -
version 0.3|2014-05-13T11:36:30+0200 (3,5,3,2,2)
2014-10-09 13:02:38,275 [main] INFO  com.graphhopper.GraphHopper - graph
LevelGraphStorage|car|MMAP_STORE_SYNC|2D|,,,,, details:edges:0(0),
nodes:0(0), name: - (0), geo:0(0),
bounds:1.7976931348623157E308,-1.7976931348623157E308,1.7976931348623157E308,-1.7976931348623157E308
2014-10-09 13:02:38,300 [main] INFO  com.graphhopper.GraphHopper - start
creating graph from ./north-america-latest.osm.pbf
2014-10-09 13:02:38,300 [main] INFO  com.graphhopper.GraphHopper - using
LevelGraphStorage|car|MMAP_STORE_SYNC|2D|,,,,, memory:totalMB:120, usedMB:8
2014-10-09 13:10:22,788 [main] INFO  graphhopper.reader.OSMReader -
500 000 (preprocess), osmIdMap:6 959 244 (86MB) totalMB:723, usedMB:297

....

2014-10-09 13:13:01,288 [main] INFO  graphhopper.reader.OSMReader -
10 000 000 (preprocess), osmIdMap:137 558 252 (1630MB) totalMB:1895,
usedMB:1726
Exception in thread "pool-1-thread-2" java.lang.OutOfMemoryError: GC
overhead limit exceeded
	at
gnu.trove.list.array.TLongArrayList.ensureCapacity(TLongArrayList.java:186)
	at gnu.trove.list.array.TLongArrayList.add(TLongArrayList.java:221)
	at
com.graphhopper.reader.pbf.PbfBlobDecoder.processWays(PbfBlobDecoder.java:286)
	at
com.graphhopper.reader.pbf.PbfBlobDecoder.processOsmPrimitives(PbfBlobDecoder.java:372)
	at
com.graphhopper.reader.pbf.PbfBlobDecoder.runAndTrapExceptions(PbfBlobDecoder.java:388)
	at com.graphhopper.reader.pbf.PbfBlobDecoder.run(PbfBlobDecoder.java:406)
....


I get a similar message for a second thread. The process stays running,
but I see no CPU activity in top.

The system is running Debian wheezy. I tested with openjdk 1.6 and 1.7.

So, how can I get this running with a large network? I'm sure there's a
command line switch or config setting somewhere that I've missed due to
my inexperience with java.

Best regards,

Nate Atkinson