[OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

>> The final event in each weekly planet dump does not fall on an  
>> even  day boundary. In the case of the most recent Oct. 22nd  
>> planet.osm, it  was necessary to experiment with hourly diffs from  
>> that day to find  that the boundary was approx. 2:00pm. Hourlies up  
>> to and including  2008102213-2008102214.osc.gz failed, hourlies  
>> after that succeeded. I  could go more granular here, checking the  
>> minute diffs as well for a  more precise breakpoint, but it seems  
>> odd that the planet dump does  not break cleanly on a midnight  
>> boundary so that it's possible to pick  up the differences moving  
>> forward.
>
> Planet dumps are not snapshots - they do not represent a consistent  
> view at any particular point in time because they take a number of  
> hours to generate, during which time new changes are constantly  
> being made to the contents of the database.

Shouldn't it be possible to ignore any changes that happen after the  
cutoff, though? I may not understand the structure of the OSM  
database, but it seems like if it supports rollbacks, then in theory  
it ought to be possible to only include things before a given  
timestamp when creating the dump file. That, or make it clear what the  
actual cutoff time is in the dumpfile.

I understand that in practice, practice is different from theory. =)


> I believe that it is supposed to be safe to apply diffs which  
> overlap with the planet dump in order to bring it to a consistent  
> state however.

This is what I would have hoped, however osm2pgsql does not appear to  
allow it. It feels like the easiest solution would be to give  
osm2pgsql a --force option, and add some explanation of timing and  
cutoffs to http://planet.openstreetmap.org/README.


> BTW I'm not sure why you CCed the OSMF board on this... I don't  
> think it needs their input at all.

Mikel Maron suggested that I cc: team@, when I spoke to him about this  
a few days ago, because it's connected to a *.openstreetmap.org service.

Thanks for your reply!

>

-mike.

----------------------------------------------------------------
michal migurski- mike at stamen.com
                  415.558.1610