[OSM-dev] Live Database Copy of Planet File

[Brett Henderson]

Matt Amos wrote:
> On Thu, Nov 5, 2009 at 12:07 PM, Emilie Laffray
> <emilie.laffray at gmail.com> wrote:
>   
>> 2009/11/5 Frederik Ramm <frederik at remote.org>
>>     
>>> Timo Juhani Lindfors wrote:
>>>       
>>>> I would like to do this too to provide a local mirror.  Last time I
>>>> tried to do this I understood that this can't be done since the 0.6
>>>> style diffs do not have enough information to create 100% identical
>>>> copy. Is this still correct? I think the problem had something to do
>>>> with changesets not being atomic.
>>>>         
>>> There are now "replication" changesets (along with a special Osmosis
>>> task, --rri, to read and assemble them) which are supposed to capture
>>> any change to the data base.
>>>       
>> Interesting. Do you have more documentation on this? I looked at the
>> reference page for Osmosis and I couldn't find it.
>> With those changesets, how do you proceed from a planet dump? Is there any
>> documentation on how to use this.
>>     
>
> i don't think there is any documentation yet, and i'm sure brett would
> be happy if someone could help out with that. :-)
>   
This is true :-)

The minute replication tasks are working and I believe they're 
reliable.  There's some additional tasks used to automate their 
downloads, and some tasks in the works to merge them into hour/day 
changesets.  They still don't contain full changeset information.  They 
do contain full history.

I've been away on holiday for five weeks, starting a new project at 
work, and now I'm moving to a new house which won't have Internet 
connectivity for at least a month.  Suffice to say I won't have much 
time to improve on the current state of affairs in the very near future.

Some of the stuff to be done includes:
* The new tasks need to be documented.
* Existing tasks need to be examined to see if the full history diffs 
(ie. changesets with potentially multiple changes to a single entity in 
the same file) work correctly.
* A new task may need to be created to condense full history diffs into 
simpler diffs to work with tasks where full history doesn't make sense 
(eg. I suspect merging tasks will have issues).
* All changeset extraction needs to be migrated onto the new servers.
* New jobs and osmosis tasks need to be created to merge minute 
replication files into larger files (ie. hour and day).
* The existing "history" extraction process needs to be continued to 
provide full history since day 1.

I'll probably get there eventually but any assistance would be appreciated.
> it's been discussed a couple of times on this ML, though:
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017390.html
> http://lists.openstreetmap.org/pipermail/dev/2009-October/017522.html
>   
Brett