[OSM-dev] osm2pgsql tile expiry freaks me out

[Steve Hill]

On Thu, 5 Nov 2009, Matt Amos wrote:

> when i tried using the osm2pgsql expiry on the minutely no-names tile
> server it slowly went out of sync. after about a week of running it
> was 2 days behind. admittedly, that EC2 server it's running on doesn't
> exactly have stellar disk performance.

To be honest, I found that just running the osm2pgsql updates really 
hammered the database, but you're right that the expiry stuff adds to that 
load.  I intend to spend some time fiddling with the postgres settings to 
see if I can tune the performance a bit.

OpenPisteMap used to manage to just about keep up (used to lag behind 
during the day and then catch up at night), but the I/O rather crippled 
the machine.  After a hard disk failed in another of my servers I had to 
move a lot of services onto the database server, which left me having to 
shut down the updates because it couldn't cope with the extra load.  I 
just got the failed server running again with 3 brand new hard disks and 
imported a new planet file onto that so I could have a go at tuning the 
postgres settings, and then 2 of the new disks failed, so I'm waiting on 
them being replaced at the moment... Not having much luck with this 
machine. :)

> yeah, the assumption is that any change to a relation is going to
> (probably) involve changes to one or more nodes and ways, so the
> relation (probably) doesn't need expiring in total.

I think that assumption is very faulty - there are plenty of tag-changes 
you can make to relations that wouldn't involve updating the component 
ways and nodes.

> indeed. and there are operations, such as reversing the way, which
> might not change the rendering at all.

The key word here is "might" :)
Unfortunately, without knowing the rendering rules, you don't know if 
reversing the way would change it or not.  In the case of one-way streets, 
it most definately would change the rendering.  The chances are that if 
someone is reversing the way, they are probably doing it for a 
rendering-related reason - i.e. they are reversing it because it has 
rendered wrong.  Of course, you also reverse ways when you merge them, but 
handling merges efficently is a whole other can of worms. :)

> indeed. it's a complex problem. there's a quick-and-dirty solution,
> but to do it properly, efficiently and accurately is very hard. i
> think jon was saying in the pub last night that diff updates and
> expiry already take up more resources than rendering tiles on yevaud.
> and that's with the quick-and-dirty solution.

The diff updates really are the killer for me - they are way more I/O 
intensive than a non-slim-mode planet import.  I'm using reasonably meaty 
machines, but the last slim-mode planet import I did took days...

-- 

  - Steve
    xmpp:steve at nexusuk.org   sip:steve at nexusuk.org   http://www.nexusuk.org/

      Servatis a periculum, servatis a maleficum - Whisper, Evanescence