[GraphHopper] Is the import really multi-thread?

[John Zhao]

Hi Peter,

If I put the pbf file in memory first, will it will be faster?

*Best Regards,*
*ZhiQiang ZHAO*

On Fri, Jul 3, 2015 at 12:15 AM, Peter <graphhopper at gmx.de> wrote:

>  Hi,
>
> Yes, more than 2 threads won't help much in practise as disc access is the
> limiting resource for import (not CPU bound!). So the default settings are
> already optimal in many cases.
>
> > 1. what is the pool 1 and pool 2?
>
> That are the standard names of the java.util.concurrent.ExecutorService I
> think. See PbfDecoder, which is from osmosis.
>
>
>  > For Europe download from geofabrik, this step take 50 minutes.
>
> You can try to put input (PBF) and output files (GH folder) on two
> separate discs, preferable SSD.
>
> Regards,
> Peter
>
>
> On 03.07.2015 01:39, John Zhao wrote:
>
> Hi,
>
>  I put parameter:
> .put("osmreader.workerThreads", Math.max(2,
> Runtime.getRuntime().availableProcessors()));
>
>  My CPU has 32 cores.
> And I saw the log as below. It looks like this worked as we thought.
> But I have a few questions:
> 1. what is the pool 1 and pool 2?
> 2. what the load of machine is pretty low, like 2-5. I thought it should
> be pretty high.
> 3. basically, this step is just convert pbf file to internal osm entity
> object?
>
>  For Europe download from geofabrik, this step take 50 minutes.
>
>  22:33:23.967 [pool-1-thread-14] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.968 [pool-1-thread-6] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.968 [pool-1-thread-23] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.969 [pool-1-thread-5] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.969 [pool-1-thread-11] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.969 [pool-1-thread-12] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.970 [pool-1-thread-22] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.970 [pool-1-thread-2] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.970 [pool-1-thread-21] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.971 [pool-1-thread-25] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
>
>  23:19:38.130 [pool-2-thread-5] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.142 [pool-2-thread-21] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.153 [pool-2-thread-13] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.164 [pool-2-thread-31] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.176 [pool-2-thread-12] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.186 [pool-2-thread-24] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.193 [pool-2-thread-4] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.203 [pool-2-thread-32] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.209 [pool-2-thread-22] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.218 [pool-2-thread-19] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.226 [pool-2-thread-3] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.238 [pool-2-thread-27] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
>  *Best Regards,*
> *ZhiQiang ZHAO*
>
>
> _______________________________________________
> GraphHopper mailing listGraphHopper at openstreetmap.orghttps://lists.openstreetmap.org/listinfo/graphhopper
>
>
>
> _______________________________________________
> GraphHopper mailing list
> GraphHopper at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/graphhopper
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20150705/845d1304/attachment.html>