[OSM-talk] Mass coastline import - Was: Server slowness

David Groom reviews at pacific-rim.net
Sun Jan 14 13:41:54 GMT 2007


My personal feeling is that unless an area is actively being worked on, then at present we shouldn't be importing coastline data until we have a hardware / software platform which can deal with the volume of data.

I completed the upload of the UK coastline, which OJW had started, I was unsure whether to do all of the Western Isles, but every now and then I came across an area someone had added a road in, and so it was hard to define a place where I should stop the coastline import, though I did not import the Shetland Isles, coast, nor that for Northern & Southern Ireland.

David Groom
  ----- Original Message ----- 
  From: Ray Booysen 
  To: talk at openstreetmap.org 
  Sent: Sunday, January 14, 2007 10:23 AM
  Subject: Re: [OSM-talk] Server slowness


  On that note, as the coastline import is pretty much automatic, should we delete the coastlines for africa until we can move to the tile based index.  Its not very important to have the coastline import currently and that should aleviate the issues for the time being.  I've spoken to the other Africa coastline importer (Firefishy) and he agrees that we should do this.  Between the two of us, large portions of the east and west of Africa have been done, but both of these can be re-imported at a later date. 

  Also, our imports finished right at the beginnning of January and we have stopped imports until these other issues can be sorted out first.

  Comments? :)

  Thanks
  Ray


  On 1/14/07, Nick Hill <nick at nickhill.co.uk> wrote:
    Looking at the disc usage for the db, 10% of the hard disc was filled with data
    between 29th Dec and 5th Jan. This represents 4Gb of data, including indexes,
    which represents a 25% increase in the total OSM data set. 

    I have taken a look at the Africa coastline. It is very detailed, and looks like
    it even details mangrove swamps etc. So it is not beyond possibility this
    increase in data set is caused by African coastline. 

    This could cause a substantial hit for queries hitting Europe. Our database
    currently narrows the index down on either lat or lon, not both. Therefore,
    look-ups for Europe may require brute force searches to remove the African 
    coastline from the indexed dataset. That would impact performance until we move
    to a tile based index.


    Nick Hill wrote:
    > It is still worrying me how the database load has shot up on 30th December. This 
    > inevitably causing slowness.
    >
    > When did the Africa coastline import run from/to?
    > How many points were involved?
    > Was there any publicity of OSM on that day which could have led to a significant 
    > increase in users? Would this result in this profile of usage ? (I think not but
    > prepared to debate it).
    >
    > Steve, did you make any changes to the API or DB set-up around that time?
    >
    >
    >
    > Andy Robinson wrote:
    >
    >> I thought it was probably the Africa Coastline stuff too. We had nearly half
    >> a million node writes a day going in earlier in the week, but as you point 
    >> out that situation has declined now. The only other thing that springs to
    >> mind is Steve's mapnick db rendering. I'm assuming that's coming off the db
    >> direct rather than using the last planet. If this is the case I presume then 
    >> it will take a while for the first run of tiles to be completed, after that
    >> the load should drop of course.
    >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20070114/2467bea7/attachment.html>


More information about the talk mailing list