[OSM-dev] Daily Diff parsing - tile expiring

[Brett Henderson]

I have something in the works which could help with this although I'm 
intending to release it along with version 0.6 updates of osmosis so 
won't be ready for a little while.

Osmosis provides a postgresql schema (called pgsql) which contains 
complete osm data without history.  It is already available with 0.5 
tasks.  It already supports application of changesets although it hasn't 
been extensively used yet.  I'm enhancing it to add an "action" column 
to node/way/relation tables.  The intent is for osmosis to mark a 
node/relation/way as modified during changeset import, then invoke a 
user extensible stored procedure where an end user can add their own 
logic (eg. to update their own extension tables).

As you've discovered, there's no simple way of doing this purely with 
xml files.  I believe the best solution will have to utilise a 
database.  The downside is that you'll need to maintain a full 
production dataset in your local database to cover the entire planet.

The solution would work something like this:
1. Create a pgsql schema.
2. Import a production snapshot (ie. the planet).
3. Begin applying changesets to keep it up to date.
4. Add a stored procedure which will look at changed ways and nodes to 
detect which tiles are impacted and write these to a new "expired_tiles" 
table.
5. Tell osmosis to trigger the stored procedure every time a changeset 
is applied (will probably be triggered by another command line argument 
to osmosis)
6. Write some code to examine the "expired_tables" and trigger 
re-rendering as appropriate.

This process could be invoked every hour or minute if desired and would 
run very quickly with short intervals due to the small size of 
incremental updates.

Roeland Douma wrote:
> Hi all,
>
> I'm working on a script to expire tiles bases on (daily) diffs. This so we can 
> re-render the tiles with changes and keep the tiles without for a longer 
> period of time. Which will allow us to have more up to date maps. And not 
> have the server do more work than necessary since tiles that have no changes 
> do not have to be re-renderd.
>
> This is easily done for nodes since they contain Lon/Lat tags. So for zoom a 
> till b the tiles can be calculated and removed.
>
> However the problem starts when ways are parsed. Since ways only contain refs 
> to nodes. So no Lat/Lon info is provided.
>
> The average daily diff contains > 700.000 node refs in way tags. 
>
> I have tried to get all the nodes (1000 nodes at the same time) from the 
> OSM-API. This solution provided me straight away with the Lon/Lat for the 
> nodes. However this took a very very long time. And it is not very nice to do 
> that on a daily basis. (also for the load on the server). 
>
> An other solution I tried was running a local copy of the database. running 
> oms2pgsql in slim-mode. However this took a long time. And when running the 
> query (with >700.000 nodes) also took a very very long time. Now of course 
> the Lat/Lon have to be parsed but that is not the problem.
>
> As you might have guessed both solutions do not work. Or at least not good 
> enough for my taste. I would like a simple (or at least fast) solution.
>
> I think that adding the Lat/Lon tag to node references in the ways is not an 
> option since it would require a huge amount of queries on the database when 
> creating the diffs. The positive side of this would that it would only have 
> to be done once.
>
> Something else would be to have some sort of bounding box tag at the ways. 
> This is not optimal in tile expiring but is better than it is done now.
>
> Anybody with idea's on this matter?
>
> Greetings,
> Roeland
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>