[OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence
Brett Henderson
brett at bretth.com
Mon Oct 27 12:50:39 GMT 2008
Frederik Ramm wrote:
> Hi,
>
> Michal Migurski wrote:
>
>> I've noticed some misalignments between the data in the dumps and the
>> osm2pgsql importer that leads to unavoidable holes in the data.
>>
>
> As TomH has already said, this is not a bug, it stems from the fact that
> the full planet export reads the "current" tables and as such is subject
> to changes that occur during the export process. (There may even be
> inconsistencies when something like this happens: Exporter dumps nodes,
> exporter starts dumping ways, user adds new node into way, new way
> version is dumped referring to new node that is not in the dump.)
>
> The daily, hourly, and minutely diffs have a clean cutoff date because
> they are taken from the history tables.
>
> Brett Henderson has offered to look into creating the dailies from
> history as well, but I don't know about the status of that.
>
Are you referring to the daily changesets? The daily, hourly and minute
changesets are all using the same osmosis-extract-mysql application and
the only difference is the interval being used. For a while it was
using a shell script for dailies that spaetz initially created and I
extended, but it was unreliable in the face of database outages and is
no longer being used. That was the reason for the switch from bzip to
gzip compression.
Or did you mean planets instead of dailies? I have a working
implementation but it is slower than the existing planet dump process so
I've never tried to introduce it. It would be faster to automate
patching of a current table planet with some changesets.
Brett
More information about the talk
mailing list