[osmosis-dev] Odd rate numbers with write-pgsql-dump

[Brett Henderson]

Hi Paul,

On 15 September 2012 18:37, Paul Norman <penorman at mac.com> wrote:

> I am using the --write-pgsql-dump in preparation for importing the planet
> into a pgsnapshot database and have noticed some oddities with ways.
>
> I am reading from the latest .pbf and using enableBboxBuilder=no
> enableLinestringBuilder=yes nodeLocationStoreType=TempFile. I tried
> InMemory
> but 16G of ram is not sufficient with the latest planet.
>
> My --log-progress task reported nodes being processed at about 250k
> nodes/second. It has now moved to the ways and the reports are interesting.
>
> It was reporting approximately 150-200 objects/second but when I manually
> take compute the ways/second from the way numbers I get figures in the
> realm
> of 5k+ objects/second. I know there may be some differences from deleted
> ways skewing the numbers, but this seems beyond what that would account
> for.
>
> It has taken approximately 20 minutes to reach way 4.5m, about 4k
> ways/second.
>
> iotop reports java reading at 5-10 M/s and writing at 150-400 K/s
> iostat reports the array with the .pbf and tempfile at 300 r/s 28 w/s 6000
> rkB/s 330 wkB/s, 90 %util.
>
> Although the progress seems okay (day or so to finish) I'm stumped by the
> objects/sec number.
>
> As an aside, about a year ago it took 5 hours to do it with InMemory but
> I'd
> have to go up to 24 GB RAM to do that again.
>

The numbers do seem a little odd, but I can't explain it.  The logic is the
same for all object types so if nodes are calculated correctly then ways
should also be calculated correctly.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20120918/20fe2069/attachment-0001.html>