[GraphHopper] Graph Storage Types and their correct initialization

Fri Apr 25 08:57:30 UTC 2014

Hey Jürgen,

>
> > do you apply some specific configuration or customization to the
> LocationIndexTree?
>
> I copied and modified mostly from OSMReader.java for my IdfReader and
> from GraphHopper.java for my own GipHopperIdf implementation.
> Modifications are mostly simplifying. I removed all parts which are
> related to ContractionHierachies for example, because I want to create
> a router which can change the graph dynamically while runtime. 

You removed for clarity? Or what was the problem?

> RAM and RAM_STORE keep crashing just when I call graph.flush() with
> the following ArrayIndexOutOfBoundsException in the RAMDataAccess
> Provider.

Another thing I forgot. See below.

> > Are you calling create() before you use the DataAccess files?
>
> No, I don't....seems that you've found my second major error here. I
> insprected OSMReader for how to do that. So when I have 1034868 nodes,
> does that mean, that I have to create GraphStorage with a bytecount of
> at least 1034868 / 50 = 20698?

The count parameter is kind of historic, so I should remove it (but
still hesitating as some implementations could benefit from such an
estimation). So just use 1000 or whatever as the DataAccess objects will
increase dynamically via appending new fixed-size segments.

> I am adding nodes to my graph while reading my Idf/Csv File. I only
> know how much nodes I have, when I am completely through. So I guess
> there is no way to make GraphStorage increase bytecount while creating
> nodes automatically and that I have to do an intermediate read of my
> file first before filling the GraphStorage (which is absolutely ok, I
> just want to know if I choose the right strategy)?

Ah, another thing I forgot, also in the javadocs! The DataAccess stuff
is kind of very low level and you need to call incCapacity everytime you
are not sure the old size fits. To avoid calling this too much you can
do similar stuff I do in GraphHopperStorage.ensureNodeIndex

> Thank you a lot for helping me out understanding...again ;)

No problem!

Peter.

>> Hey Jürgen,
>>
>> how much RAM do you assign for this process? (the -Xmx setting)
>> What version of GraphHopper do you use?
>> And do you apply some specific configuration or customization to the
>> LocationIndexTree?
>>
>> > I already am aware, that I should not flush() the graph when I use
>> RAM or RAM_STORE (because everything is kept in Memory I guess?).
>> > When I use MMAP, the reader flushes after every 100000 items which
>> were read and after finishing reading.
>>
>> I think flush is only necessary at the end even for MMAP.
>>
>>
>> > I already am aware, that I should not flush() the graph when I use
>> RAM or RAM_STORE (because everything is kept in Memory I guess?).
>>
>> You should still flush at the end. When you use 'RAM' then simply
>> nothing happens.
>>
>> But when you use RAM_STORE 'flush' writes to disk. The next time you
>> start GH it then can avoid parsing and it will just load the data
>> from disk into memory
>>
>>
>> > Exception in thread "main" java.lang.NullPointerException
>> >    at
>> com.graphhopper.storage.MMapDataAccess.newByteBuffer(MMapDataAccess.java:176)
>>
>> Are you calling create() before you use the DataAccess files?
>> Or what is 'null' on line 176 - would you point to the github source
>> for it?
>>
>> > First I have to say, I'm sorry, that I am not really totally
>> focused on this,
>>
>> No problem. Good to have some other usecases to make it robust and
>> more userfriendly.
>>
>> > initialize GraphStorage
>>
>> Please see the javadocs of GraphHopperStorage:
>> Life cycle: (1) object creation, (2) configuration via setters &
>> getters, (3) create or loadExisting, (4) usage, (5) flush, (6) close
>>
>> Regards,
>> Peter.
>>
>>
>>> Hi Peter,
>>>
>>> I can't get my head around storage types in graphhopper, and how to
>>> treat them correctly. I have my own little Idf File reader which
>>> works perfectly with small graphs. Although there is still much work
>>> to do (writing tests, refactor some classes, actually test) it reads
>>> my idf mockup graphs and routes correctly then.
>>>
>>> But when I take a larger amount of data (Graph of austria with
>>> approx. 1 million nodes and 1.2 million links) my 4GB 4 Core Linux
>>> Notebook runs out of memory, when I use DAType.RAM_STORE
>>>
>>> 2014-04-23 20:57:20,221 [main] INFO  reader.idf.IdfReader - Starting
>>> to read nodes!
>>> 2014-04-23 20:57:24,281 [main] INFO  reader.idf.IdfReader - Read
>>> 1034868 records which equals 1034868 rows as expected
>>> 2014-04-23 20:57:24,281 [main] INFO  reader.idf.IdfReader - Graph
>>> has 1034868 nodes.
>>> 2014-04-23 20:57:24,281 [main] INFO  reader.idf.IdfReader - Starting
>>> to read links!
>>> 2014-04-23 20:57:36,024 [main] INFO  reader.idf.IdfReader - Read
>>> 1207004 records which equals 1207004 rows as expected
>>> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead
>>> limit exceeded
>>>     at
>>> com.graphhopper.storage.index.LocationIndexTree$InMemTreeEntry.<init>(LocationIndexTree.java:844)
>>>     at
>>> com.graphhopper.storage.index.LocationIndexTree$InMemConstructionIndex.addNode(LocationIndexTree.java:428)
>>>     at
>>> com.graphhopper.storage.index.LocationIndexTree$InMemConstructionIndex.addNode(LocationIndexTree.java:433)
>>>     at
>>> com.graphhopper.storage.index.LocationIndexTree$InMemConstructionIndex.addNode(LocationIndexTree.java:433)
>>>     at
>>> com.graphhopper.storage.index.LocationIndexTree$InMemConstructionIndex.addNode(LocationIndexTree.java:433)
>>>
>>> When I use DAType.MMAP I get a NullPointerException immediately
>>> after inserting the very first node, while creating the
>>> "newByteBuffer" in MMapDataAccess.java:
>>> 2014-04-23 21:18:27,264 [main] INFO  reader.idf.IdfReader - Starting
>>> to read nodes!
>>> Exception in thread "main" java.lang.NullPointerException
>>>     at
>>> com.graphhopper.storage.MMapDataAccess.newByteBuffer(MMapDataAccess.java:176)
>>>     at
>>> com.graphhopper.storage.MMapDataAccess.mapIt(MMapDataAccess.java:150)
>>>     at
>>> com.graphhopper.storage.MMapDataAccess.incCapacity(MMapDataAccess.java:103)
>>>     at
>>> com.graphhopper.storage.GraphHopperStorage.ensureNodeIndex(GraphHopperStorage.java:261)
>>>     at
>>> com.graphhopper.storage.GraphHopperStorage.setNode(GraphHopperStorage.java:232)
>>>     at
>>> com.graphhopper.reader.idf.IdfReader.loadGraph(IdfReader.java:237)
>>>     at
>>> com.graphhopper.reader.idf.IdfReader.doIdf2Graph(IdfReader.java:102)
>>>     at com.graphhopper.GipHopperIdf.importINTREST(GipHopperIdf.java:200)
>>>     at com.graphhopper.GipHopperIdf.process(GipHopperIdf.java:175)
>>>     at com.graphhopper.GipHopperIdf.importOrLoad(GipHopperIdf.java:159)
>>>     at com.graphhopper.GipHopperIdf.main(GipHopperIdf.java:42)
>>>
>>>
>>> I already am aware, that I should not flush() the graph when I use
>>> RAM or RAM_STORE (because everything is kept in Memory I guess?).
>>> When I use MMAP, the reader flushes after every 100000 items which
>>> were read and after finishing reading. In this example where I load
>>> all Nodes and Links from Austria, no graph files are created in the
>>> graph-location folder.
>>>
>>> First I have to say, I'm sorry, that I am not really totally focused
>>> on this, because I develop on this reader really as a hobby project,
>>> while doing ten others. But for now I did a lot of debugging and
>>> research but I think I am completely stuck at the moment, which is
>>> possibly a result of my incomplete understanding of how to
>>> initialize a GraphStorage correctly.
>>>
>>> Any hints appreciated.
>>>
>>> best regards,
>>>
>>> Jürgen

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20140425/9c9b9bbc/attachment.html>