[GraphHopper] Is the import really multi-thread?

[Peter]

Hi,

Yes, more than 2 threads won't help much in practise as disc access is
the limiting resource for import (not CPU bound!). So the default
settings are already optimal in many cases.

> 1. what is the pool 1 and pool 2?

That are the standard names of the java.util.concurrent.ExecutorService
I think. See PbfDecoder, which is from osmosis.


> For Europe download from geofabrik, this step take 50 minutes.

You can try to put input (PBF) and output files (GH folder) on two
separate discs, preferable SSD.

Regards,
Peter

On 03.07.2015 01:39, John Zhao wrote:
> Hi,
>
> I put parameter:
> .put("osmreader.workerThreads", Math.max(2,
> Runtime.getRuntime().availableProcessors()));
>
> My CPU has 32 cores.
> And I saw the log as below. It looks like this worked as we thought.
> But I have a few questions:
> 1. what is the pool 1 and pool 2?
> 2. what the load of machine is pretty low, like 2-5. I thought it
> should be pretty high.
> 3. basically, this step is just convert pbf file to internal osm
> entity object?
>
> For Europe download from geofabrik, this step take 50 minutes.
>
> 22:33:23.967 [pool-1-thread-14] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.968 [pool-1-thread-6] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.968 [pool-1-thread-23] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.969 [pool-1-thread-5] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.969 [pool-1-thread-11] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.969 [pool-1-thread-12] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.970 [pool-1-thread-22] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.970 [pool-1-thread-2] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.970 [pool-1-thread-21] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 22:33:23.971 [pool-1-thread-25] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
>
> 23:19:38.130 [pool-2-thread-5] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.142 [pool-2-thread-21] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.153 [pool-2-thread-13] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.164 [pool-2-thread-31] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.176 [pool-2-thread-12] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.186 [pool-2-thread-24] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.193 [pool-2-thread-4] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.203 [pool-2-thread-32] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.209 [pool-2-thread-22] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.218 [pool-2-thread-19] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.226 [pool-2-thread-3] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> 23:19:38.238 [pool-2-thread-27] DEBUG c.g.reader.pbf.PbfBlobDecoder -
> Processing OSM primitive group.
> *Best Regards,*
> *ZhiQiang ZHAO*
>
>
> _______________________________________________
> GraphHopper mailing list
> GraphHopper at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/graphhopper

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/graphhopper/attachments/20150703/f7fb8520/attachment.html>