[osmosis-dev] using osmosis to import full history planet

[Martijn van Exel]

Hi Peter,

On 3/8/2011 1:03 PM, Peter Körner wrote:
> Am 08.03.2011 12:47, schrieb Martijn van Exel:
>> Do you mean to say there is a history .jar in that build? I don't see
>> one. Or do you mean that I can use the history plugin with that build?
> You can use the history jar you build during the last days with this
> version of osmosis.
>

So do I just copy the osmosis-history.jar to the 0.37 osmosis? Does it 
go into lib/default?

>>>
>>> Please be warned: I never imported more than a few nodes into a history
>>> database. The used node-stores are not able to handle more than some
>>> thousand nodes.
>>>
>>> If you use the plan writing without linestring or bbox builder they
>>> should not be used and you should be able to import a much bigger set of
>>> data.
>>
>> I do want the linestring option in the end, the BBOX I can probably do
>> without. Let me first try it without the linestring builder and see if
>> that works. What kind of work would it take to include the linestring
>> builder with a larger dataset? Maybe I can try and pool some resources
>> here.
>
> The main problem is that we need to fiddle out which version of a node
> correlates to which version of the way. Because the way does not have a
> direct reference to the node-version, it's required to compare the
> timestamps.

It would make sense to me if this was being dealt with during the 
generation of the full history planet file. But things being as they 
are, timestamp-checks are unavoidable.
>
> This is done using the HistoryNodeStore interface. It's only
> implementation ExampleHistoryNodeStore uses two instances of
> java.util.Map to store the nodes. This is not efficient enough to run
> with larger datasets but it's well tested and good enough for developing.

Is it possible to mimic osm2pgsql's --slim behaviour, create temporary 
tables in postgres to hold all the nodes and work from those instead of 
memory?

> A quick look shows that the linestring-builder should be in a usable
> state despite the storage problems. The last thing I was working on was
> the minor version builder, that aims to create a "half version" of a way
> each time one of its nodes got modified. This is somehow experimental.

OK, if I can get it to work I will see how it behaves with more data.
Thanks again,
Martijn

-- 
Martijn van Exel
Senior Researcher
-------------------------------------
Geodan S&R
President Kennedylaan 1
1079 MB Amsterdam (NL)
-------------------------------------
Tel: +31 (0)20 - 5711 318
Fax: +31 (0)20 - 5711 333
-------------------------------------
E-mail: martijn at geodan.nl
Website: www.geodan.nl
KvK-nummer: 33 247475
Disclaimer: www.geodan.nl/disclaimer
-------------------------------------