[osmosis-dev] Reading OSM History dumps

[Brett Henderson]

On Wed, Aug 25, 2010 at 11:14 PM, Peter Körner <osm-lists at mazdermind.de>wrote:

> Brett, the pgsql tasks currently write (in COPY mode) all data to temp
> files first. The process seems to be
>
> PlanetFile -> NodeStoreTempFile -> CopyFormatTempFile -> PgsqlCopyImport
>
> in osm2pgsql the copy data is pushed to pgsql via unix pipes (5 or 6 COPY
> transactions running at the same time in different connections). This
> approach skips the CopyFormatTempFile stage. Is there any special reason
> this approach isn't used in the pgsnapshot package?
>

Not too sure now :-)  I think it was the simplest way to share code between
both the --write-pgsql-dump task and what was then the --fast-write-pgsql
(now simply --write-pgsql) task.

In practice the COPY file creation and loading is fairly fast.  The biggest
downside is the extra disk space.  The slowest parts of the whole process
are the way geometry creation, index building, and the CLUSTER statements
(in the newest schema).  On relatively low-end hardware it takes many days
to import an entire planet, only a small part of which is the COPY
processing.

In most cases I create the COPY files using --write-pgsql-dump and load them
via the provided load script so that I can better monitor progress and
resume if processing is interrupted.

In short it just hasn't been a high priority to change it.

While I'm on the topic, I've mostly completed the changes to the schema
now.  Performance is drastically improved over the old version for bounding
box query processing.  The --read-pgsql --dataset-bounding-box task
combination would previously take approximately an hour to retrieve a 1x1
degree box in a populated area, now it is down to around 5 minutes due to
far better disk layout.  The biggest downside is that the table clustering
takes a long time during initial database creation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20100825/d7cc5230/attachment.html>