[OSM-dev] osm2pgsql, zoom range for -e option?

marqqs at gmx.eu marqqs at gmx.eu
Wed Aug 10 15:40:49 BST 2011


My new patch cares about issue 1 and 2:



$ diff -C 5 -p expire-tiles_old.c expire-tiles.c
*** expire-tiles_old.c	2011-05-21 15:13:49.000000000 +0200
--- expire-tiles.c	2011-08-10 16:23:05.000000000 +0200
*************** static int _mark_tile(struct tile ** tre
*** 105,114 ****
--- 105,146 ----
   */
  static int mark_tile(struct tile ** tree_head, int x, int y, int zoom) {
  	return _mark_tile(tree_head, x, y, zoom, 0);
  }
  
+ #if 1
+ static void output_dirty_tile(FILE * outfile, int x, int y, int zoom, int min_zoom) {
+   // writes a tile into output file, including all subsequent tiles
+   // with higher zoom levels;
+   int x_max, y_max;
+   int tile_size;
+ 
+   tile_size= 1;
+   while (zoom <= Options->expire_tiles_zoom) {
+     if(zoom>=min_zoom) {
+       x_max = x + tile_size;
+       while (x < x_max) {
+         y_max = y + tile_size;
+         while (y < y_max) {
+ 		      if ((outcount++ % 1000) == 0) {
+ 			      fprintf(stderr, "\rWriting dirty tile list (%ik)", outcount / 1000);
+ 			      fflush(stderr);
+ 		      }
+ 		      fprintf(outfile, "%i/%i/%i\n", zoom, x, y);
+           y++;
+         }
+         y-= tile_size;
+         x++;
+       }
+       x-= tile_size;
+     }
+     zoom++;
+     x<<= 1; y<<= 1;
+     tile_size<<= 1;
+   }
+ }
+ #else
  static void output_dirty_tile(FILE * outfile, int x, int y, int zoom, int min_zoom) {
  	int	y_min;
  	int	x_iter;
  	int	y_iter;
  	int	x_max;
*************** static void output_dirty_tile(FILE * out
*** 131,148 ****
--- 163,191 ----
  			}
  			fprintf(outfile, "%i/%i/%i\n", out_zoom, x_iter, y_iter);
  		}
  	}
  }
+ #endif
  
  static void _output_and_destroy_tree(FILE * outfile, struct tile * tree, int x, int y, int this_zoom, int min_zoom) {
  	int	sub_x = x << 1;
  	int	sub_y = y << 1;
  	FILE *	ofile;
  
  	if (! tree) return;
  
+ #if 1
+   if(this_zoom >= min_zoom) {
+     if ((outcount++ % 1000) == 0) {
+       fprintf(stderr, "\rWriting dirty tile list (%ik)", outcount / 1000);
+       fflush(stderr);
+     }
+     fprintf(outfile, "%i/%i/%i\n", this_zoom, x, y);
+   }
+ #endif
+ 
  	ofile = outfile;
  	if ((tree->complete[0][0]) && outfile) {
  		output_dirty_tile(outfile, sub_x + 0, sub_y + 0, this_zoom + 1, min_zoom);
  		ofile = NULL;
  	}





If you change the two "#if 1" to "#if 0", the module will return to its old behaviour.

Markus

-------- Original-Nachricht --------
> Datum: Wed, 10 Aug 2011 15:16:22 +0200
> Von: marqqs at gmx.eu
> An: Igor Podolskiy <igor.podolskiy at vwi-stuttgart.de>, dev at openstreetmap.org
> Betreff: Re: [OSM-dev] osm2pgsql, zoom range for -e option?

> Hi Igor,
> 
> thanks!
> 
> Do your python functions recalculate all the missing tile paths?
> 
> Example:
> The dirty_tiles file contains just one tile:
> 
> 16/100/100
> 
> Would the output of your function produce the following result?
> 
> 16/100/100
> 17/200/200
> 17/200/201
> 17/201/200
> 17/201/201
> 18/400/400
> 18/400/401
> 18/400/402
> 18/400/403
> 18/401/400
> 18/401/401
> 18/401/402
> 18/401/403
> 18/402/400
> 18/402/401
> 18/402/402
> 18/402/403
> 18/403/400
> 18/403/401
> 18/403/402
> 18/403/403
> 15/50/50
> 14/25/25
> 13/12/12
> 12/6/6
> 11/3/3
> 
> Markus
> 
> -------- Original-Nachricht --------
> > Datum: Wed, 10 Aug 2011 14:26:17 +0200
> > Von: Igor Podolskiy <igor.podolskiy at vwi-stuttgart.de>
> > An: dev at openstreetmap.org
> > Betreff: Re: [OSM-dev] osm2pgsql, zoom range for -e option?
> 
> > Hi,
> > 
> > > The expired-tiles list which is written by osm2pgsql does not contain
> > any redundant information. This is intended that way because he wanted
> to
> > save memory: the quad-tree in the algorithm should not grow too much.
> > >
> > > For this reason there are two groups of tiles excluded from the list:
> > >
> > > 1. Next lower zoom levels
> > > For example, if a tile at zoom level 18 is considered as dirty, you
> will
> > need to render the related tile at level 17 too. The list will only
> > contain the level-18 tile.
> > >
> > > 2. Next higher zoom levels
> > > If four neighboring tiles at zoom level 18 are dirty and they fit into
> a
> > single level-17 tile, only the level-17 tile is marked as dirty, the
> four
> > level-18 tiles are not.
> > >
> > > Of course I can understand the need to save memory while the
> > expired-tiles list is created but in my opinion this list should be
> written as a
> > complete list of expired tiles...
> > aha, so it's not a bug, it's a feature... good to know.
> > 
> > I worked around this issue (didn't have time to dig further in 
> > expire-tiles.c then) with the following Python function:
> > 
> > def expire_lower_zoom(filename, lowest_zoom):
> >      for line in open(filename, 'r'):
> >          z, x, y = [int(i) for i in line.strip().split('/')]
> >          for zz in xrange(z, lowest_zoom-1, -1):
> >              yield (zz, x, y)
> >              x, y = x / 2, y / 2
> > 
> > Or, avoiding duplicates by storing everything in a hashset (in memory):
> > 
> > def expire_lower_zoom_no_dup(filename, lowest_zoom):
> >      result = set()
> >      for line in open(filename, 'r'):
> >          z, x, y = [int(i) for i in line.strip().split('/')]
> >          for zz in xrange(z, lowest_zoom, -1):
> >              result.add((zz, x, y))
> >              x, y = x / 2, y / 2
> >      return result
> > 
> > You could do the same for a stdin stream or any Python iterable, for 
> > that matter, the procedure for invalidating higher zoom levels is almost
> > the same. You can even avoid writing the function for higher zoom levels
> > by passing "-e 17" (or whatever your highest zoomlevel is) to osm2pgsql 
> > to make it only expire tiles on your highest zoomlevel, and the above 
> > function will then compute all the lower zoom levels. It worked for me 
> > for a while now.
> > 
> > Or am I missing something?
> > 
> > Hope that helps
> > Igor
> > 
> > _______________________________________________
> > dev mailing list
> > dev at openstreetmap.org
> > http://lists.openstreetmap.org/listinfo/dev
> 
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev



More information about the dev mailing list