[OSM-dev] speeding up loading an OSM dump into PostGIS?

Kai Krueger kakrueger at gmail.com
Sat Dec 17 00:51:52 GMT 2011


On 01/-10/-28163 12:59 PM, sly (sylvain letuffe) wrote:
> Hi,
> 
>> We're working
>> hard on getting the relevant hardware in place to start trialling this
>> out, but it's a big project.
> 
> Many thanks for the insight
>  
>> The original topic was about replication for rendering, so a comment
>> on that 
> 
> Whooops, I haven't read that thread far enough in the past to discover I'm off 
> topic.
> I do agree that livre replication for rendering has few to no interests, and I 
> was talking about live replication in order to load balance editing read 
> calls.
> 
> I'll start a new topic about that later but we are in the process of acquiring 
> 3 huge servers for our activities in France, one of wich might be used for 
> API ro calls and I'll conduct a few tests to check if a 3 minutes lag behing 
> main db doesn't have significant edits drawbacks.
> Final goal is having faster and less limited ro api for editing
>  

There was some talk a while back about having sub-minutely diffs. If I
remember correctly, there is no technical reason to not offer
replication-diffs at a granularity of less than a minute. E.g. every 30
seconds or perhaps even every 10 seconds. However, so far there hasn't
really been any compelling use cases for sub minutely diffs. If you
really are going to set up a ro mirror for editing, perhaps this idea
can be revisited.

However, I would also suspect that a lag of 2 - 3 minutes would be
mostly fine anyway. The current work flow is to download some data, then
edit for a while after which you upload changes. The "edit for a while"
stage is probably typically longer than 2 - 3 minutes anyway so by the
time you upload you are working with "old data".

One would presumably have to adapt JOSM and Potlatch slightly anyway to
be able to handle a secondary read only mirror. By default they would
download data from the ro-mirror. In the case of conflict, uploads, or
an explicit update request by the user, it would still use the main api
for reads to make sure it has the most up-to-date data.

Kai






More information about the dev mailing list