[osmosis-dev] Inquiries about "Minutely Mapnik"

[Frederik Ramm]

Hi,

Maxime Petazzoni wrote:
> We are looking for solutions on how to solve this problem, and it seems
> like reducing the delta of the updates to hourly increments (or even
> minutely increments) with Osmosis + osm2pgsql could help.

It is unlikely to speed things up but at least you can chop your updates 
into smaller bits. I highly recommend to stop processing map requests 
while you do updates. Since Maposmatic has this queuing mechanism 
anyway, you can afford to mix operations. The beauty of the minutely 
updates with Osmosis is that you can run them at any time, so you can 
delay updating if the site is very busy rendering, or do one update 
after each render, or so.

> I indeed
> expect applying a minute diff to be significantly fast, and that the sum
> of these processings be smaller than what it would take for a daily
> diff.

Unlikely; the larger diff will probably give you more efficient use of 
the cache, and as you will be using the --simc flag with Osmosis (won't 
you?) you will only process one change to an object which has changed 
multiple times in the whole interval.

> From your experience, can you confirm that this would be the case, and
> that applying minute diffs won't take longer than a minute, or hourly
> diffs longer than an hour (otherwise it would be impossible to stay
> up-to-date with the database and we would start falling behind)?

It is really very much dependent on what hardware you run this on. I 
have a modern, 12 GB, quad-core machine with dual cheap SATA disks 
configured into RAID0 and on that applying diffs takes, on average, 50% 
of the time the diff is for, but may occasionally also be much less or 
much more. The nice thing about the minutely update process is that it 
has a reasonable capability of healing itself, so if applying an 
one-hour diff takes two hours once in a while then the process is likely 
to get back to an equilibrium.

> The "Minutely Mapnik" wiki page on the OSM wiki
> (http://wiki.openstreetmap.org/wiki/Minutely_Mapnik) seems to be a good
> starting point to get this new technique of keeping our database updated
> going. But I would like to try it first on my machine. Importing the
> full planet.osm file would take forever though, so I'd like to try this
> on a single country. Is it possible to use the osmosis replication data
> on a subset of the planet?

You can only get worldwide updates. osm2pgsql copes ok when you add 
worldwide diffs to a reduced database; you can even specify a bbox 
filter on the command line to ignore stuff that is outside. This doesn't 
work perfectly (e.g. if a node is moved from inside the bbox to outside, 
osm2pgsql will discard that information because it looks like it 
pertains to something outside, thus an old version of the node will 
remain) but should be ok for testing.

Bye
Frederik