[OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

Brett Henderson brett at bretth.com
Mon Oct 27 12:50:39 GMT 2008


Frederik Ramm wrote:
> Hi,
>
> Michal Migurski wrote:
>   
>> I've noticed some misalignments between the data in the dumps and the  
>> osm2pgsql importer that leads to unavoidable holes in the data.
>>     
>
> As TomH has already said, this is not a bug, it stems from the fact that 
> the full planet export reads the "current" tables and as such is subject 
> to changes that occur during the export process. (There may even be 
> inconsistencies when something like this happens: Exporter dumps nodes, 
> exporter starts dumping ways, user adds new node into way, new way 
> version is dumped referring to new node that is not in the dump.)
>
> The daily, hourly, and minutely diffs have a clean cutoff date because 
> they are taken from the history tables.
>
> Brett Henderson has offered to look into creating the dailies from 
> history as well, but I don't know about the status of that.
>   
Are you referring to the daily changesets?  The daily, hourly and minute 
changesets are all using the same osmosis-extract-mysql application and 
the only difference is the interval being used.  For a while it was 
using a shell script for dailies that spaetz initially created and I 
extended, but it was unreliable in the face of database outages and is 
no longer being used.  That was the reason for the switch from bzip to 
gzip compression.

Or did you mean planets instead of dailies?  I have a working 
implementation but it is slower than the existing planet dump process so 
I've never tried to introduce it.  It would be faster to automate 
patching of a current table planet with some changesets.

Brett





More information about the talk mailing list